Articles in the technology category

Last spring I set up a Raspberry Pi to record humming birds at my hummingbird feeder, compile the recorded h264 files into an mp4 and upload it to YouTube.

I’ve written about that process before here, here, and here.

This post is a bit of documentation to remind myself about how to connect the PIR sensor to the Raspberry Pi so I don’t forget.

When I went to set it up this year, it appeared like the PIR sensor wasn’t working. It would start the video capture, but it wouldn’t end it.

I tried a couple of different things (including the purchase of some new PIR sensors) but none of them seemed to work. I was worried that the heat from the early bit of summer last year had maybe fried my little Pi.

But no, it turns out that the link I was using as the basis for my project had a diagram AND a description about how to connect the PIR.

I had assumed that the diagram was correct and that I didn’t need to read the description, but it turns out I did BECAUSE the diagram had the connections set up in a way that didn’t line up with the PIR sensor(s) I have.

In the Parent Detector PIR sensor the connectors are (1) Power, (2) Ground, (3) Out

In my PIR sensor the connectors are (1) Power, (2) Out, (3) Ground.

This meant that the power was getting to the PIR sensor, but there was no way to send the trip because the signal was being sent to the Ground.

Anyway, the morale of the story is, pictures are nice, but reading may save you some time (and money) in the long run.

Posted by ryan on Sat 10 November 2018

It’s time for Kings Hockey! A couple of years ago Emily and I I decided to be Hockey fans. This hasn’t really meant anything except that we picked a team (the Kings) and ‘rooted’ for them (i.e. talked sh*t* to our hockey friends), looked up their position in the standings, and basically said, “Umm ... yeah, we’re hockey fans.”

When the 2018 baseball season ended, and with the lack of interest in the NFL (or the NBA) Emily and I decided to actually focus on the NHL. Step 1 in becoming a Kings fan is watching the games. To that end we got a subscription to NHL Center Ice and have committed to watching the games.

Step 2 is getting notified of when the games are on. To accomplish this I added the games to our family calendar, and decided to use what I learned writing my ITFDB program and write one for the Kings.

For the Dodgers I had to create a CSV file and read it’s contents. Fortunately, the NHL as a sweet API that I could use. This also gave me an opportunity to use an API for the first time!

The API is relatively straight forward and has some really good documentation so using it wasn’t too challenging.

import requests
from sense_hat import SenseHat
from datetime import datetime
import pytz
from dateutil.relativedelta import relativedelta



def main(team_id):
    sense = SenseHat()

    local_tz = pytz.timezone('America/Los_Angeles')
    utc_now = pytz.utc.localize(datetime.utcnow())
    now = utc_now.astimezone(local_tz)

    url = 'https://statsapi.web.nhl.com/api/v1/schedule?teamId={}'.format(team_id)
    r = requests.get(url)

    total_games = r.json().get('totalGames')

    for i in range(total_games):
        game_time = (r.json().get('dates')[i].get('games')[0].get('gameDate'))
        away_team = (r.json().get('dates')[i].get('games')[0].get('teams').get('away').get('team').get('name'))
        home_team = (r.json().get('dates')[i].get('games')[0].get('teams').get('home').get('team').get('name'))
        away_team_id = (r.json().get('dates')[i].get('games')[0].get('teams').get('away').get('team').get('id'))
        home_team_id = (r.json().get('dates')[i].get('games')[0].get('teams').get('home').get('team').get('id'))
        game_time = datetime.strptime(game_time, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=pytz.utc).astimezone(local_tz)
        minute_diff = relativedelta(now, game_time).minutes
        hour_diff = relativedelta(now, game_time).hours
        day_diff = relativedelta(now, game_time).days
        month_diff = relativedelta(now, game_time).months
        game_time_hour = str(game_time.hour)
        game_time_minute = '0'+str(game_time.minute)
        game_time = game_time_hour+":"+game_time_minute[-2:]
        away_record = return_record(away_team_id)
        home_record = return_record(home_team_id)
        if month_diff == 0 and day_diff == 0 and hour_diff == 0 and 0 >= minute_diff >= -10:
            if home_team_id == team_id:
                msg = 'The {} ({}) will be playing the {} ({}) at {}'.format(home_team, home_record, away_team, away_record ,game_time)
            else:
                msg = 'The {} ({}) will be playing at the {} ({}) at {}'.format(home_team, home_record, away_team, away_record ,game_time)
            sense.show_message(msg, scroll_speed=0.05)


def return_record(team_id):
    standings_url = 'https://statsapi.web.nhl.com/api/v1/teams/{}/stats'.format(team_id)
    r = requests.get(standings_url)
    wins = (r.json().get('stats')[0].get('splits')[0].get('stat').get('wins'))
    losses = (r.json().get('stats')[0].get('splits')[0].get('stat').get('losses'))
    otl = (r.json().get('stats')[0].get('splits')[0].get('stat').get('ot'))
    record = str(wins)+'-'+str(losses)+'-'+str(otl)
    return record


if __name__ == '__main__':
    main(26) # This is the code for the LA Kings; the ID can be found here: https://statsapi.web.nhl.com/api/v1/teams/

The part that was the most interesting for me was getting the opponent name and then the record for both the opponent and the Kings. Since this is live data it allows the records to be updated which I couldn’t do (easily) with the Dodgers programs (hey MLB ... anytime you want to have a free API I’m ready!).

Anyway, it was super fun and on November 6 I had the opportunity to actually see it work:

<iframe src="https://www.youtube.com/embed/AzdLSrA8wvU" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe>

I really like doing fun little projects like this.

Posted by ryan on Fri 09 November 2018

New  Watch

The first week¶

I've been rocking a series 2 Apple Watch for about 18 months. I timed my purchase just right to not get a series 3 when it went on sale (🤦🏻‍♂️). When the series 4 was released I decided that I wanted to get one, but was a bit too slow (and tired) to stay up and order one at launch.

This meant that I didn't get my new Apple Watch until last Saturday (nearly5 weeks later). I wanted to write down my thoughts on the  Watch and what it's meant for me. I won't go into specs and details, just what I've found that I liked and didn't like.

The Good¶

Holy crap is it fast. I mean, like really fast. I've never had a watch that responded like this (before my series 2 I had a series 0).

It reacts when I want it to, so much so that I'm sometimes not prepared. It reminds me of the transition from Touch ID Gen 1 to Touch ID Gen 2. I really appreciate how fast everything comes up. When I start an activity, it’s there (no more waiting like on Series 2). When I want to pair with my AirPods … it’s there and ready to go.

I also really like how much thinner it is and the increase in size. At first I thought it was ‘monstrous’ but now I’m trying to figure out how I ever lived with 2 fewer millimeters.

I also decided to get the Cellular Version just in case. It was a bit more expensive, and I probably won’t end up using it past the free trial I got, but it’s nice to know that I can have it if I need it. I haven’t had a chance to use it (yet) but hopefully I’ll get a chance here soon.

The Bad¶

So far, nothing has stuck me as being ‘bad’. It’s the first Apple Watch I’ve had that’s really exceeded my expectations in terms of performance and sheer joy that I get out of using it.

Conclusion¶

Overall I love the Series 4  Watch. It doesn’t do anything different than the Series 2 that I had (except I can make phone calls without my phone if I need to) but oh my is it fast! If someone is on a Series 2 and is wondering if jumping to the Series 4 is worth it … it totally is.

Posted by ryan on Sat 03 November 2018

Every once in a while I get a wild hair and decide that I need to ‘clean up’ my directories. This never ends well and I almost always mess up something, but I still do it.

Why? I’m not sure, except that I forget that I’ll screw it up. 🤦‍♂️

Anyway, on a Saturday morning when I had nothing but time I decided that I’d move my PyCharm directory from /Users/ryan/PyCharm to /Users/ryan/Documents/PyCharm for no other reason than because.

I proceeded to use the command line to move the folder

mv /Users/ryan/PyCharm/ /Users/ryan/Documents/PyCharm/

Nothing too big, right. Just a simple file movement.

Not so much. I then tried to open a project in PyCharm and it promptly freaked out. Since I use virtual environments for my Python Project AND they tend to have paths that reference where they exist, suddenly ALL of my virtual environments were kind of just gone.

Whoops!

OK. No big deal. I just undid my move

mv /Users/ryan/Documents/PyCharm/ /Users/ryan/PyCharm

That should fix me up, right?

Well, mostly. I had to re-register the virtual environments and reinstall all of the packages in my projects (mostly not a big deal with PyCharm) but holy crap it was scary. I thought I had hosed my entire set of projects (not that I have anything that’s critical … but still).

Anyway, this is mostly a note to myself.

The next time you get a wild hair to move stuff around, just keep it where it is. There’s no reason for it (unless there is).

But seriously, ask yourself first, “If I don’t move this what will happen?” If the answer is anything less than “Something awful” go watch a baseball game, or go to the pool, or write some code. Don’t mess with your environment unless you really want to spend a couple of hours unmasking it up!

Posted by ryan on Sun 12 August 2018

I have many meetings that I go to in any given day. One of the things that I’d been struggling with was being able to keep track of what I needed to do after a meeting and/or documenting certain types of meetings more effectively.

I have been using a Workflow I created a couple of years ago to get the pertinent details of a meeting into Drafts. I spoke about updating that workflow to incorporate drafts 5 here.

Once I was able to get the information into Drafts 5 a new opportunity arose. I was able to run a Workflow in Drafts!

I decided that getting the information into Drafts was great, but I needed a good way to get it out.

There were two sections in the Draft that I decided I could leverage to help:

1. Actions
2. Notes

Broadly speaking there are 3 types of meetings I go to:

1. Daily Standup aka Scrum
2. One-on-One with direct reports or my manager
3. General Meetings

Categorizing the meetings helped me to create Draft Actions that run Workflows for each meeting type.

Scrum¶

This workflow runs through the Actions of the Draft and adds each one to OmniFocus in a Project called Scrum with a Tag of Work. The due date set for these tasks is noon of the same day. My goal is to have the items that come from Scrum totally processed by noon of that day and for 80% of them I can. Some actions are more involved, but having them in OmniFocus helps me to make sure that they get taken care of.

It also creates a calendar meeting for the next business day with my Scrum template and lets me know which team member will start that next day.

One-on-One¶

This workflow runs similarly to the Scrum workflow. It adds the Action items to OmniFocus with a due date of noon the same day, tagged with Work and in the One-on-One Project.

Instead of creating a calendar meeting for the next business day at 8:30 it appends items from the Notes section to a Dropbox file. The Dropbox path is predefined, but the name of the file matches the name of the person I met with (luckily I don’t have 2 Tom’s reporting to me).

General Meetings¶

This is the simplest workflow. It adds all of the items under actions to OmniFocus with a due date of noon, project of Meeting Follow Up and Tag of Work.

After the Actions are run from Drafts the notes are archived in Drafts.

I’m toying with the idea of archiving the notes from these meetings into Dropbox, but I’m not sure that it gets me anything ... so I haven’t really looked at it too deeply.

Workflow links¶

The links for each of the workflows can be found here:

Parse Scrum Notes

Parse One-on-One Notes

Parse Meeting Notes

Posted by ryan on Thu 07 June 2018

A Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

Uses include:

1. data cleaning and transformation
2. numerical simulation
3. statistical modeling
4. data visualization
5. machine learning
6. and other stuff

I’ve been interested in how to set up a Jupyter Notebook on my Linode server for a while, but kept running into a roadblock (either mental or technical I’m not really sure).

Then I came across this ‘sweet’ solution to get them set up athttp://blog.lerner.co.il/five-minute-guide-setting-jupyter-notebook-server/

My main issue was what I needed to to do keep the Jupyter Notebook running once I disconnected from command line. The solution above gave me what I needed to solve that problem

nohup jupyter notebook

nohup allows you to disconnect from the terminal but keeps the command running in the background (which is exactly what I wanted).

The next thing I wanted to do was to have the jupyter notebook server run from a directory that wasn’t my home directory.

To do this was way easier than I thought. You just run nohup jupyter notebook from the directory you want to run it from.

The last thing to do was to make sure that the notebook would start up with a server reboot. For that I wrote a shell script

# change to correct directory
cd /home/ryan/jupyter

nohup jupyter notebook &> /home/ryan/output.log

The last command is a slight modification of the line from above. I really wanted the output to get directed to a file that wasn’t in the directory that the Jupyter notebook would be running from. Not any reason (that I know of anyway) … I just didn’t like the nohup.out file in the working directory.

Anyway, I now have a running Jupyter Notebook at http://python.ryancheley.com:8888¹

1. I’d like to update this to be running from a port other than 8888 AND I’d like to have it on SSL, but one thing at a time! ↩︎

Posted by ryan on Sun 27 May 2018

Drafts is a productivity app created by Greg Pierce (\@AgileTortoise).

I’ve loved and used Drafts 4 every day for the last several years. I loved it so much I even contributed to the Tip Jar Greg had in the app. Seriously, it’s an amazing app. If you haven’t downloaded it already you totally should.

Recently, Greg released Drafts 5. With this new version comes a new Business Model as well. Instead of a single pay (and hope people ‘tip’ you) he’s converted to a subscription model.

I signed up for the free week and didn’t have a real opportunity to use it before my free week was converted into a pay week but I’ve no regrets. I like what Greg does and want him to keep updating his app so that I can get the benefits of it once I have a real chance to dive in.

Part of the reason I wasn’t able to really use the new version is the way that I primarily use Drafts. I have a WorkFlow that takes a meeting on my work calendar and allows me to take notes about that meetings.

It’s one of the most useful productivity tools I have during my morning standup meetings with my team, and it’s useful for the other (sometimes endless) meetings that I go to.

With the release of Drafts 5 I was not longer able to use both Drafts 5 AND my workflow, so I needed to update my workflow.

With Drafts 4 it was just one of the built in Apps. Because Drafts 5 limits some of the functionality unless you have the PRO version I don’t think that Workflow will be updated to include Drafts 5 like it did Drafts 4.

Once I realized that AND since I’m paying for the app I figured I’d need to update my Workflow instead of waiting and hoping that Workflow would be updated to include Drafts 5.

In order to make the update I had to look for URL Scheme for Drafts 5 ... but I couldn’t really find one. I assumed that Drafts 5 URL Scheme would be the same as Drafts 4 (I was right) and made various attempts at getting a copy of the Workflow to work with Drafts 5.

This is the section of the workflow that needs to be updated:

Since Drafts 5 isn’t included in the Built in Apps I was going to need to pass a URL and open the app.

This would require 3 separate steps in Workflow

1. Convert Text into URL Encoded string
2. Prepend the URL Scheme for creating a new draft to the URL Encoded String
3. Open the URL

This basically means that 1 step is now replaced with 3 ... but hey, that’s the price of progress must be paid!

Both the Drafts 4 and Drafts 5 versions of these workflows are available.

If you enjoy them, hit me up in the comments or let me know on Twitter \@ryancheley!

Posted by ryan on Sat 19 May 2018

Logging into a remote server is a drag. Needing to remember the password (or get it from 1Password); needing to remember the IP address of the remote server. Ugh.

It’d be so much easier if I could just

ssh username@servername

and get into the server.

And it turns out, you can. You just need to do two simple things.

Simple thing the first: Update the hosts file on your local computer to map the IP address to a memorable name.¶

The hosts file is located at /etc/hosts (at least on *nix based systems).

Go to the hosts file in your favorite editor … my current favorite editor for simple stuff like this is vim.

Once there, add the IP address you don’t want to have to remember, and then a name that you will remember. For example:

67.176.220.115    easytoremembername

One thing to keep in mind, you’ll already have some entries in this file. Don’t mess with them. Leave them there. Seriously … it’ll be better for everyone if you do.

Simple thing the second: Generate a public-private key and share the public key with the remote server¶

From the terminal run the command ssh-keygen -t rsa. This will generate a public and private key. You will be asked for a location to save the keys to. The default (on MacOS) is /Users/username/.ssh/id_rsa. I tend to accept the default (no reason not to) and leave the passphrase blank (this means you won’t have to enter a password which is what we’re looking for in the first place!)

Next, we copy the public key to the host(s) you want to access using the command

ssh-copy-id <username>@<hostname>

for example:

ssh-copy-id pi@rpicamera

The first time you do this you will get a message asking you if you’re sure you want to do this. Type in yes and you’re good to go.

One thing to note, doing this updates the file known_hosts. If, for some reason, the server you are ssh-ing to needs to be rebuilt (i.e. you have to keep destroying your Digital Ocean Ubuntu server because you can’t get the static files to be served properly for your Django project) then you need to go to the known_hosts file and remove the entry for that known host.

When you do that you’ll be asked about the identity of the server (again). Just say yes and you’re good to go.

If you forget that step then when you try to ssh into the server you get a nasty looking error message saying that the server identities don’t match and you can’t proceed.

Posted by ryan on Sat 05 May 2018

Several weeks ago in Cronjob Redux I wrote that I had finally gotten Cron to automate the entire process of compiling the h264 files into an mp4 and uploading it to YouTube.

I hadn’t. And it took the better part of the last 2 weeks to figure out what the heck was going on.

Part of what I wrote before was correct. I wasn’t able to read the client_secrets.json file and that was leading to an error.

I was not correct on the creation of the create_mp4.sh though.

The reason I got it to run automatically that night was because I had, in my testing, created the create_mp4.sh and when cron ran my run_script.sh it was able to use what was already there.

The next night when it ran, the create_mp4.sh was already there, but the h264 files that were referenced in it weren’t. This lead to no video being uploaded and me being confused.

The issue was that cron was unable to run the part of the script that generates the script to create the mp4 file.

I’m close to having a fix for that, but for now I did the most inelegant thing possible. I broke up the script in cron so it looks like this:

00 06 * * * /home/pi/Documents/python_projects/cleanup.sh
10 19 * * * /home/pi/Documents/python_projects/create_script_01.sh
11 19 * * * /home/pi/Documents/python_projects/create_script_02.sh >> $HOME/Documents/python_projects/create_mp4.sh 2>&1
12 19 * * * /home/pi/Documents/python_projects/create_script_03.sh
13 19 * * * /home/pi/Documents/python_projects/run_script.sh

At 6am every morning the cleanup.sh runs and removes the h264 files, the mp4 file and the create_mp4.sh script

At 7:10pm the ‘header’ for the create_mp4.sh runs. At 7:11pm the ‘body’ for create_mp4.sh runs. At 7:12pm the ‘footer’ for create_mp4.sh runs.

Finally at 7:13pm the run_script.sh compiles the h264 files into an mp4 and uploads it to YouTube.

Last night while I was at a School Board meeting the whole process ran on it’s own. I was super pumped when I checked my YouTube channel and saw that the May 1 hummingbird video was there and I didn’t have to do anything.

Posted by ryan on Wed 02 May 2018

After days of trying to figure this out, I finally got the video to upload via a cronjob.

There were 2 issues.

Issue the first¶

Finally found the issue. Original script from YouTube developers guidehad this:

CLIENT_SECRETS_FILE = "client_secrets.json"

And then a couple of lines later, this:

% os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

When crontab would run the script it would run from a path that wasn’t where the CLIENT_SECRETS_FILE file was and so a message would be displayed:

WARNING: Please configure OAuth 2.0

To make this sample run you will need to populate the client_secrets.json file
found at:

  %s

with information from the Developers Console
https://console.developers.google.com/

For more information about the client_secrets.json file format, please visit:
https://developers.google.com/api-client-library/python/guide/aaa_client_secrets

What I needed to do was to update the CLIENT_SECRETS_FILE to be the whole path so that it could always find the file.

A simple change:

CLIENT_SECRETS_FILE  = os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

Issue the second¶

When the create_mp4.sh script would run it was reading all of the h264 files from the directory where they lived BUT they were attempting to output the mp4 file to / which it didn’t have permission to write to.

This was failing silently (I’m still not sure how I could have caught the error). Since there was no mp4 file to upload that script was failing (though it was true that the location of the CLIENT_SECRETS_FILE was an issue).

What I needed to do was change the create_mp4.sh file so that when the MP4Box command output the mp4 file to the proper directory. The script went from this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " hummingbird.mp4") > create_mp4.sh

To this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " /home/pi/Documents/python_projects/hummingbird.mp4") > /home/pi/Documents/python_projects/create_mp4.sh

The last bit /home/pi/Documents/python_projects/create_mp4.sh may not be necessary but I’m not taking any chances.

The video posted tonight is the first one that was completely automatic!

Now … if I could just figure out how to automatically fill up my hummingbird feeder.

Posted by ryan on Fri 20 April 2018