Headless RaspberryPI Setup – No Monitor, Keyboard or Ethernet

ruibmLeave a comment

window-1-197

And the day to revive my good old RaspberryPi Version 1 Model B has come.

As it always happens, I end up finding myself chasing tails trying to set it up without having to plug and keyboard, display or ethernet to it. I just expect that it can magically plug into my local WiFi and start the ssh server so I can remotely login.

So this time around, and as a future note to self, here’s how I got it to work with all the little quirks and weirdnesses that ended up wasting about 1hour of my life until it was up and running:

1. Download the Raspbian OS image.

This can easily be downloaded from the Raspbian website here.

At the time of this post, I ended up downloading Raspbian Stretch. Initially I went for the Lite version and in the end I used the Desktop version in the hope I wouldn’t have to ‘apt-get install‘ as much stuff.

Not a very scientific measurement, but the Desktop version seemed to boot up and connect to the WiFi a lot quicker than the Lite version. I didn’t have a monitor plugged so I honestly have no idea what was going on.

2. Flash the image onto your SD/MicroSD card.

For this you can use the Etcher.io app. It’s a very clean cross-platform application to flash images onto SD card. I was honestly very very happy to see someone finally took the time to implement a cross platform flasher. The days of ‘dd‘ seem to be over. 🙂

3. Configure the /boot partition.

Whether you are on a Windows or on a Mac, once you’ve flashed the SD card your computer should be able to mount a partition called ‘boot‘.

3.1. To tell Raspbian to start the ssh server once it boots, you will need to create an empty file called ‘ssh’ inside the ‘boot’ partition.

3.2 In order to give Raspbian the WiFi credentials and SSID you want it to connect to you will need to create a file called ‘wpa_supplicant.conf’ in the same ‘boot’ partition. Here’s what you should write inside this file. Please make sure to update $SSID and $PASSWORD with the name and password of your WiFi:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
  ssid="$SSID"
  psk="$PASSWORD"
  key_mgmt=WPA-PSK
}

4. Boot your RaspberryPi.

Unmount the SD card from your computer and insert it into your RaspberryPi.

The  RaspberryPi Version 1 Model B I own does not have onboard WiFi so I purchased a WiFi USB dongle. Make sure that is connected as well before you turn on the RaspberryPi.

And finally, power on your RaspberryPi. This is the step where you will need to be very patient. When I booted the Lite image, it took minutes (>10min) until it finally connected to the WiFi and was available to be ssh’ed into.

5. When is it ready to be used?

I usually just leave a ping command running in a tight loop in my terminal and wait until the RaspberryPi starts returning valid pings. Here’s the bash command I use:

$ ping raspberrypi | while read pong; do echo "$(date): $pong"; done

And voilá, you should be good to go. Now you just need to run the following command type ‘raspberry’ as the password:

$ ssh pi@raspberrypi

6. So how did I waste 1h trying to set this up?

Well, the RaspberryPi can be quite unforgiving and gives you basically no feedback.

Once you turn it on, the red LED for Power stays fixed and the SD card led flickers around a bit but then switches off… and that’s it… No feedback whether an error happened, whether it’s stuck waiting for input, whether it’s absolutely fine, just nothing…

In my first few attempts to boot it and connect it to WiFi I waited about 5min on each attempt to see it connect and when no new MAC address was registering with the router I just assumed something went wrong, read a little more on the Internet about potentially problems and retry again with a different tweak.

Eventually, I saw that it worked because I mostly gave up and went to bed. And after 10min+ the damn thing finally lit up the WiFi LED in the USB dongle and it managed to connect. So basically, coincidence. I was already planning to plug it via ethernet the next day, or, if all failed, just plug the damn HDMI+Keyboard and connect to it directly.

For some reason the Raspbian Desktop image boots up and turns WiFi a lot more quickly (a couple of minutes), so I will now on stick to that.

7. Useful links and documentation.

Here are a couple of links that helped me out understanding a little more what was going on. I also went through a couple of reddit threads but at this point I stand no chance in getting those links back.

Upgrading Jawbone Mini Jambox

ruibm3 Comments

91tRcTlun7L._SL1500_This is a post for the unlikely event that you find yourself owning an old ‘Jawbone Jambox Mini‘ **and** you want to change the voice pack back to original/default, after you spent a couple of years using the Japanese one – a voice pack you don’t really understand but thought it would be entertaining to listen to a new voice when you turn on the speaker.

So Jawbone went under in 2017 (https://en.wikipedia.org/wiki/Jawbone_(company)) so not an awful lot of hopes they will provide any new software or update the old one.

To make it a little gloomier, their website was completely taken down and they never really open sourced the code for their hardware nor did they post to some third-party website the last state of all the software for their products.

Luckily, before Jawbone’s website was completely taken down, Robert decided to capture all the latest software and posted it elsewhere with instructions on how to update. You can find all the goodies here:

NOW A WORD OF ADVICE!!

I use a macbook and I can confirm that if you try to use the Mac Jawbone Updater 2.2.5 to update/change the language pack of your Jawbone Mini Jambox, the upgrade will fail and you will end up with a bricked speaker. 😦

The good news is that this is recoverable. Here’s how to do it:

  1. Unplug the Mini Jambox.
  2. Power down the Mini Jambox.
  3. Press and hold the Play/Pause button (circular button on top).
  4. While still holding the button, plug the Mini Jambox to a power source.
  5. Voilá, your Jambox should now be in recovery mode and you can flash it/upgrade it with a new language package.

In order to properly upgrade the language pack of your Jawbone Mini Jambox you should be running on Windows and use the Windows “Jawbone Updater.exe”. I did try initially doing this via VMWare Fusion, running a Windows 8.1 VM inside my Mac OS X. That does not work.

You need to run the Jawbone Updater software from a native Windows operating system (no VMs involved).

And that’s it, your Jawbone Mini Jambox is good to go for a little longer.

Rotating Videos and Preserving Metadata

ruibmLeave a comment

So this is a pet peeve that’s been biting me for a sometime.

Often when you take a picture with you phone or digital camera, the camera rotation sensor (gyroscope?) gets it wrong and ends  up taking a picture that is sideways (ie, rotated +/- 90 degrees).
Surely enough you can use your phone’s photo library app to rotate it and job done, sanity restored.

However, the really really annoying thing is when you camera gets the orientation of a video wrong. Rotating a video is just not a functionality that the photo library offers off the shelf. That’s because rotating a video involves actually re-enconding it from the scratch. Some nice video players (eg, VLC) allow you to rotate while playing the video, but this is fiddly and hard to remember to do every time you play the video.

So in any case, here are a few bash lines that I cobbled together to rotate a batch of videos trying to preserve as much of the metadata information as possible.

for i in $( find *.mov ); do
  IN=$i OUT=${IN}.m4v && \
  ffmpeg -i ${IN} -c:a copy -map_metadata 0:s:0  -vf "transpose=1"  -c:v libx264 -crf 23 -preset medium ${OUT} && \
  exiftool -tagsfromfile ${IN} -makernotes -make -model ${OUT} && \
  exiftool ${OUT} && \
  rm *_original
done

NOTE: To run these you will need to have ffmpeg and exiftool installed in your system.

Bash Shell $PS1 Configuration

ruibmLeave a comment

After a recent talk with Jim Meyering I’ve decided to finally organise a bit my .bashrc and all my .dot_files in general.

So first and foremost important change, track all .dot_files in some for of version control system. I’m right now using Mercurial powered by BitBucket. Git is also a great choice. Go with whatever you are comfortable with, just make sure you don’t lose your precious configs and that you can easily synchronise all you unix/linux/bsd boxes effortlessly.

The other big take for this talk was to make sure your $PS1 shell prompt gives you the right information. Two key things that are absolute gold to have:

  1. The branch/bookmark you are currently in if you are in a VCS directory.
  2. The exit code of the previous command if it returned an error (different from 0).

Here’s how my current $PS1 looks like:

.bashrc $PS1

And this is my current .bashrc $PS1 configuration:

############################################
# Decorate $PS1
############################################
function __get_vcs {
local path=`pwd`
while true; do
if [[ -d "${path}/.hg" ]]; then
echo "mercurial"
return
elif [[ -d "${path}/.git" ]]; then
echo "git"
return
elif [[ "${path}" = "/" ]]; then
echo "none"
return
fi
path=`cd ${path}/../ && pwd`
done
}

function __get_vcs_branch {
vcs="$(__get_vcs)"
out=''
case "$vcs" in
"mercurial")
out=`hg id -b`
out=" (hg:${out})"
;;
"git")
out=`git rev-parse --abbrev-ref HEAD`
out=" (git:${out})"
;;
*)
out=''
;;
esac
echo "$out"
}

function __get_exit_code {
local code="$?"
local msg=''
if [ $code != 0 ]; then
msg="[${code}] "
fi
echo "$msg"
}

red='\[\033[01;31m\]'
green='\[\033[00;32m\]'
purple='\[\033[01;35m\]'
yellow='\[\033[01;33m\]'
blue='\[\033[01;34m\]'
black='\[\033[00m\]'
export PS1="${red}\$(__get_exit_code)${blue}\t ${green}\W${purple}\$(__get_vcs_branch)${blue} \$${black} "

Finally, here are some other things that could be interesting to display:

  • \u – Username
  • \h – Hostname
  • \w – Full path of the current working directory

PS: Word of advice, it’s very easy to get carried away and try to add ‘the world information’ to your $PS1. Up to you what you value the most.

Cave Story

ruibmLeave a comment

cave_storyI thought I’d write a quick one about the amazing game Cave Story turned out to be. Takes you back to the old school 2D platformers with quite a high dosage of difficulty and at times the mandatory frustration. Having finished it in easy mode I can say that the final bosses in other modes must take insane amounts of mastery.

But anyway, the point of this post was not so much about the game but the fact Daisuke Amaya single handedly wrote the story, the music, the cinematics, the graphics and coded it all – this all over the course of 5 years working on his spare/free time (as he held a fulltime job). 5 years is a long time to be hammering at the same project so I can only imagine it took a lot of perseverance (read stubbornness) to stick with the project and execute it to completion. Hats off to Daisuke. 🙂

Japan Funny Signs

ruibm

Japan has really awesome sites to visit, great food and the people are extremely friendly and helpful. In the spirit of keeping tourists even more entertained, Japanese signs introduce subtle ‘easter eggs’ into their English translations. Here are some I found mildly amusing.

We’ll start with this very friendly ash cloud on a sign stuck to the floor right before a pedestrian crossing. Almost makes you wanna smoke to get your cutie genie cloud out of your bottle/cigarette. Japan Funny Pics - 01

This should probably also be enforced at local theater’s. I was also informed that “handy phone” is how germans refer to cell phones – funny the Japanese would go to a German to translate into English.

Japan Funny Pics - 02

This one is also commonly known as “palmier” but I like it they are descriptive of their pastries. Unfortunately often the show cased pastries have nothing to do with what’s inside the box.

Japan Funny Pics - 03

Were it not for the foggy cloudy weather and a gorgeous view of Mount Fuji would present itself at the top of this cable car – I mean ‘rorpeway’.

Japan Funny Pics - 04

Now that we are clear that we are taking the ‘rorpeway’ service we should get ready to head to the ‘platfome’.

Japan Funny Pics - 05

And now that we are done with our tour and the weather was so miserable that Mount Fuji was nowhere to be seen we shall go back down the ‘rorpeway’ onto our favorite ‘platfome’ and it’s promised to be ‘wicket’. That or we are soon to attend a cricket match.

Japan Funny Pics - 06

And just as we exit, Sherlock Holmes gives us a heads up by telling us to “watch our steps” – any wrong move and we are back in jail.

Japan Funny Pics - 07

So here I am right after hotel checkout. The hotel employee kindly asked me to confirm all the expenses at the hotel and once I acknowledged and paid it he professionally stamped the receipt as ‘Recieved by credit’ – a whole new level having hotel stamps with typos! One can never lower his guard.

Japan Funny Pics - 08

So the Bullet Train – Shinkansen seemed to have a rather complete and long manual on how to deal with emergencies… and they almost made it through – the final word let them down from perfection.

Japan Funny Pics - 09

Time for some snack and I have to say they went rather bold at this pastry shop using such expensive tricky words to spell – and they almost made it through…

Japan Funny Pics - 10

At a temple in Kyoto, I was already well aware of the challenges that the word ‘ticket’ presents so it came as no surprise to me the following ‘tiket’ price sign nailed to one of the temples walls.

Japan Funny Pics - 11

And free I indeed felt.

Japan Funny Pics - 12

This time around I was at a huge mall in Tokyo Station and when I glimpsed such a big sign with an English translation I instantly knew they were asking for trouble… (“Reataurant”, “Clothng”)

Japan Funny Pics - 13 Japan Funny Pics - 14

Finally, I had to dispose my coke bottle somewhere and what better place to do so than the dust boxes at the entrance of the mall. It must be a cultural thing but I often visit the mall without a hoover or a broom so I used these boxes to what was most convenient at the time.

Japan Funny Pics - 15

 

Still a couple more days until the trip is done so keep tuned for more pics… 🙂

Hadoop/Hive – Writing a Custom SerDe (Part 1)

ruibm1 Comment

(Special thanks to Denny Lee for reviewing this)

Apache Hadoop is an open source project composed of several software solutions for distributed computing. One interesting component is Apache Hive that lets one leverage the MapReducing powers of Hadoop through the simple interface of HiveQL language. This language is basically a SQL lookalike that triggers MapReduces for operations that are highly distributable over huge datasets.

A common hurdle once a company decides to use Hadoop and Hive is: “How do we make Hadoop understand our data formats.”. This is where the Hadoop SerDe terminology kicks in. SerDe is nothing but a short form for Serialization/Deserialization. Hadoop makes available quite a few Java interfaces in its API to allow users to write their very own data format readers and writers.

Step by step, one can make Hadoop and Hive understand new data formats by:
1) Writing format readers and writers in Java that call Hadoop APIs.
2) Packaging all code in a java library – eg., MySerDe.jar.
3) Adding the jar to the Hadoop installation and configuration files.
4) Creating Hive tables and explicitly set the input format, the output format and the row format.

Before diving into the Hadoop API and Java code it’s important to explain what really needs to be implemented. For concisiveness of terms I shall refer to row as the individual unit of information that will be processed. In the case of good old days SQL databases this indeed maps to a table row. However our datasource can be something as simple as Apache logs. In that case, a row would be a single log line. Other storage types might take complex message formats like Protobuf Messages, Thrift Structs, etc… For any of these, think of the top-level struct as our row. What’s common between them all is that inside each row there will be sub-fields (columns), and those will have specific types like integer, string, double, map, …

So going back to our SerDe implemention, the first thing that will be required is the row deserializer/serializer (RowSerDe). This java class will be in charge of mapping our row structure into Hive’s row structure. Let’s say each of our rows corresponds to a java class (ExampleCustomRow) with the three fields:

  1. int id;
  2. string description;
  3. byte[] payload;

The RowSerDe should be able to mirror this row class and their properties into Hive’s ObjectInspector interface. For each of our types it’ll find and return the equivalent type in the Hive API. Here’s the output of our RowSerDe for this example:

  1. int id -> JavaIntObjectInspector
  2. string description -> JavaStringObjectInspector
  3. byte[] payload -> JavaBinaryObjectInspector
  4. class ExampleCustomRow -> StructObjectInspector

In the example above, the row structure is very flat but for examples where our class contains others classes and so forth, the RowSerDe needs to be able to recursively reflect the whole structure into Hive API objects.

Once we have a way of mapping our rows into hadoop rows, we need to provide a way for hadoop to read our files or databases that contain multiple rows and extract them one by one. This is done via de Input and Output format APIs. A simple format for storing multiple rows in a file would be separating them by newline characters (like comma separated files do). An Input reader in this case would need to know how to read a byte stream and single out byte arrays of individual lines that would later be fed into to our custom SerDe class.

As you can probably imagine by now, the Output writer needs to do exactly the opposite: it receives the bytes that corresponds to each line and it knows how to append them and separate them (by newline characters) in the output byte stream.

How an Hadoop MapReduce interacts with a custom SerDe for Hive.
How an Hadoop MapReduce interacts with a custom SerDe for Hive.

Summarizing, in order to implement a complete SerDe one needs to implement:
1) The Hive Serde interface (contains both the Serializer and Deserializer interfaces).
2) Implement the InputFormat interface and the OutputFormat interface.

In the next post I’ll take a deep dive into the actual Hadoop/Hive APIs and Java code.
(Two years have gone by and I unfortunately never got round to writing anything else. Probably, anything I would write now would be outdated so I would encourage anyone who has questions to try to ping me directly or just ask directly the the hive community)

Counting lines in files

ruibmLeave a comment

Here’s silly (and long) Python script I wrote a while back to count the number of lines in files recursively through a dir. (I guess I was fed up with ‘find ../ -iname “*cpp” | xargs wc -l” or just utterly bored or using Windows…)

#!/usr/bin/python
#
# Usage:
#   ./count_lines.py [path_to_dir_or_file]

__author__ = "Rui Barbosa Martins (ruibmartins@gmail.com)"

import optparse
import os.path
import sys

def getNumberOfLines(filePath):
  fp = file(filePath, "r");
  numberOfLines = len(fp.readlines())
  fp.close()
  return numberOfLines

def visit((fileTypesIncluded, files), dirname, names):
  for f in names:    
    filePath = os.path.join(dirname, f)
    if (fileTypesIncluded == ["*"] or 
        filter(lambda fe: f.endswith(fe), fileTypesIncluded)) and
        os.path.isfile(filePath):
      files[filePath] = getNumberOfLines(filePath)

def main(startPath, fileTypesIncluded):
  if not os.path.exists(startPath):
    print "Error: Path [%s] does not exist." % (startPath)
    sys.exit(-1)
  elif os.path.isfile(startPath):
    files = {startPath: getNumberOfLines(startPath)}
  else:
    fileTypes = "|".join(fileTypesIncluded)
    absPath = os.path.abspath(startPath)
    print "Searching for extensions '%s' in '%s'." % (fileTypes, absPath)    
    files = {}
    os.path.walk(startPath, visit, (fileTypesIncluded, files))

  keys = files.keys()
  keys.sort()
  total = 0
  longestLength = 0
  groups = {}

  for key in keys:
    if longestLength < len(key):
      longestLength = len(key)      

  for key in keys:
    spaces = " " * (longestLength - len(key))
    print "%s%s %d" % (key, spaces , files[key])
    total = total + files[key]

  if len(keys) > 1:
    print "Total lines of code: %d" % (total)

def parseArgs():
  usage = "Usage: %prog [Options] FILE_OR_DIRECTORY"
  parser = optparse.OptionParser(usage=usage)
  parser.add_option("-f", "--file_extensions", dest="file_extensions",
                    help="File extensions to count in. (comma separated)", 
                    metavar="FILE_TYPES",
                    default=[".*"])
  (options, args) = parser.parse_args()
  fe = options.file_extensions
  if not args:
    path = "."
  elif len(args) == 1:
    path = args[0]
  else:
    parser.print_help()
    print "Error: Too many paths provided. Only one expected."
    sys.exit(0)

  if type(fe) == str:
    fe = map(lambda s: s.strip(), fe.split(","))
  if not fe or filter(lambda s: "*" in s, fe):
    fe = ["*"]
  return path, fe

if __name__ == "__main__":
  main(*parseArgs())

Como reparar o seu computador

ruibmLeave a comment

Há uns tempos atrás o venerável jovem do xkcd teve um rasgo de brilhantismo ao descrever no post abaixo a típica situação pela qual o comum informático passa vezes sem conta.

Como achei que era uma perda enorme tão bom diagrama não chegar à comunidade não informática portuguesa, onde o diagrama seria claramente mais útil resolvi traduzi-lo para Português. Aqui  vai o link para uma imagem com resolução aceitável para imprimirem e colocarem na parede do vosso escritório.

apoio_tecnico

SimplePhoto version 1.0.0

ruibmLeave a comment

simplephoto-150x150Finally, after almost a year battling away with wxWidgets, GraphicsMagick, gcc, Visual Studio, CppUnitLite, … I finally get to release version 1.0.0 of SimplePhoto.

SimplePhoto is a batch processing application for images. For the time being it allows image format, image dimensions and groupings.

My main focus for this application was to make it’s memory footprint as little as possible. It’s been implemented in C++ and on Windows it takes around 5MB of memory when running. There are still loads of features to implement but I really wanted to get this out there ASAP. Next step is to open source it (probably hosting it at code.google.com).

Have fun!