Archive for September, 2010

September 29, 2010 0

Written In The Stars: Dawn Mission Space Craft

By in Recreation

Asteroid Belt certificate

DARCY COOPER STEVENSON

Your name is traveling aboard the Dawn spacecraft
on its mission to the Asteroid Belt.
Nasa logo and Dawn Mission URL

September 29, 2010 0

Logical Volume Management

By in Information Technology, Linux

Logical volumes are logical constructs relating to mass storage devices. Logical Volume Management systems make several physical disks seem like one or more large storage areas.

I wrote a presentation handout on LVM as seen in the linked document entitled, Logical Volume Management.

September 29, 2010 0

Server Side Spam Filtering

By in Information Technology, Linux, Programming

I ran across a document that I wrote in 2003 outlining the spam filtering process in Postfix for a client I had at that time. I had fun working with this client, for reasons not the least of which as they were happy to let me release my documents under the GPL v2 license.

I have the PDF document linked here.

September 28, 2010 0

Update Databases Dynamically

By in Artificial Nerual Networks, Programming

I recently had to work with data for importation into a database that provided several challenges. Here’s the raw output:

Calculating indicator AroonUp[25, {I:Prices HIGH}, {I:Prices LOW}] …
AroonUp[25, {I:Prices HIGH}, {I:Prices LOW}][2010-05-03 09:01:00] = 68.0000
AroonDown[25, {I:Prices HIGH}, {I:Prices LOW}][2010-05-03 09:01:00] = 12.0000
AroonOsc[25, {I:Prices HIGH}, {I:Prices LOW}][2010-05-03 09:01:00] = 56.0000

About 45,000 or so of these records. My table should look like this:

|————————————————————————————————-|

|     date        |     time   | aroonup25 | aroondown25 | aroonosc25 |

| 2010-05-03  | 09:01:00 | 68.0000    |     12.0000      |  56.0000    |

|————————————————————————————————-|

I don’t know ahead of time which indicator the file will specify (i.e. up25, down25, osc25) nor do I know whether or not the record is already in the database.

Putting the correct values is especially acute as a) I need to ensure the date & time corresponds with the correct values (using UNIX’s ‘cut’ just won’t cut it, pun intended) and b) over the course of literally “hundreds of thousands” of records, performance becomes a real issue.

I wrote a script (listed below) that coordinates the indicator type, date, time, and value. The first step is to parse the input file to an output into database syntax. Here’s the intermediate step’s output:

INSERT INTO ARG (aroonup25, date, time) VALUES (‘68.0000’, ‘2010-05-03′, ’09:01:00′) ON DUPLICATE KEY UPDATE aroonup25=’68.0000’;
INSERT INTO ARG (aroondown25, date, time) VALUES (‘12.0000’, ‘2010-05-03′, ’09:01:00′) ON DUPLICATE KEY UPDATE aroondown25=’12.0000’;
INSERT INTO ARG (aroonosc25, date, time) VALUES (‘56.0000’, ‘2010-05-03′, ’09:01:00′) ON DUPLICATE KEY UPDATE aroonosc25=’56.0000’;

Now that I’ve created a file that contains the correct database syntax, all that’s needed is to import it into the database:

$ cat ./aroon.sql | mysql minute -u cstevens -p

While the script seems simple enough it’s actually a cornerstone to building a sophisticated method for storing indicator (in this case) data.

Here’s the script:

#!/bin/ksh

parse_file=./aroon.txt
tmp_parse_file=./tmp.txt
sql_file=./aroon.sql
table_name=$security

database_table=minute

#break down the input file
for i in 25 #25 is the period length of the indicator (for future expansion to other time lengths)
do
cat $parse_file | while read line
do
#grab ’em (ticks), parse, and convert to lower case
awk -F'[‘ -v counter=$i ‘{print $1’counter’,$3}’| sed s/’] =’//g | tr [:upper:] [:lower:] > $tmp_parse_file
done
done

if [ -f $sql_file ]
then
rm $sql_file
fi

# create sql file output
cat $tmp_parse_file | while read indicator date time value
do
print INSERT INTO ARG ‘(‘$indicator’, date, time)’ VALUES ‘(‘”‘”$value”‘”, “‘”$date”‘”, “‘”$time”‘”‘)’ ON DUPLICATE KEY UPDATE $indicator=”‘”$value”‘”‘;’ >> $sql_file
done

Note that this script contains a couple of “hard coded” variables. I consider this poor programming practice. In this case, the hard coded variables are actual variables in my master script. I will run through the necessary changes as I import this code into the code tree.

September 25, 2010 0

Creating New Business Models

By in Information Technology

The other night I met a new friend. Gordon and I got to know each other. Invariably I asked him what he did for a living.

He told me that he translated Japanese text for businesses. He explained that he received manuscripts from a few firms for his translations.

“What kinds of manuscripts to you often get?” I asked. “It varies but I’ll often get business proposals. I think my favorites are newspaper articles, those are the most interesting.”

“Really?” I thought.”Say,” I said, thinking in ‘early-often’ mode, “what would happen if you scanned all the major Japanese newspapers daily and translated all the articles that are relavent to your clients?”

“Now that would be something,” Gordon replied, “I don’t think I would do it for myself but my PR firms would love a service like that. They could outsource all the newspaper work to me.”

In technical terms, we’re talking about ‘push‘ technology. In this instance, Gordon’s IT system scans the major newspapers each day for items that are in his PR firm’s interests, translates the content, then pushes the translation to the PR firm for delivery. The advantages of this model are:

  • Steady revenue flow through contracts
  • Access, even though indirectly, to potentially hundreds of clients
  • Fewer clients to work with

Then, of course, we talk about the idea of an automated translation algorithm to help out with the translations.

September 23, 2010 0

Spammers and the Mothers Who Love Them

By in Information Technology

Spam messages purported to come from a person on your contact list is becoming a problem. You receive a message, look at the sender and think, “oh, I received a message from my friend.” The trouble is that instead of a friendly greeting you receive a (sometimes offensive) advertisement from a mass-mailer, a.k.a. a “spammer.”

This happens because either a) the sender’s mail was infected by malware on their PC or b) the message was “spoofed” with your friend’s email address by the spammer using an infected computer that happens to have your friend in their contact list.

Unfortunately, the end result is the same, often too is the solution: the sender needs to change she or he’s email address.
To track down the problem, first look in your “Sent Items” folder to see if your account actually sent the message. This happens as infected computers (the sender’s laptop, etc.) was “taken hostage” and is now sending emails out to recipients in the sender’s contact list. If you (the sender) find that these messages were sent by your computer then you’ll need to disinfect your system.

Second, make sure your login credentials are strong: change your password to at least seven characters with at least a number (not at the end) and a special character (%&*#) or two. Look to change your login’s secret questions and answers also.

Third, as mentioned above, scan and disinfect your machine if you find that the messages were sent from your account.

Finally, it is best to change your address. Let everyone on your contact list know that you’re now using a new address.

For parting thoughts I was to write something like, “overall, spammers cost us millions… etc.” but in the final analysis, it’s more than that–spammers are a nuisance, they are sometimes even hurtful. They need to be dealt with and stopped.

September 22, 2010 2

The Game of Go at Powell’s

By in Recreation

Powell’s Books hosts the Portland Go Club every Tuesday night beginning at 6:00 p.m. Doug and I headed down to have a look. Two long tables were arrayed with Go boards with players sitting opposite of each other. You could hear the stones tap and shuffle on the boards as you walked past. I estimate about 16 games goin on.

Doug and I went around to each of the games and said our “hello’s.” We then took a look at some books. I’m impressed by how much Doug knows not just about the game itself but also of the game’s people and rich history. I felt like I was on a guided tour spanning several hundred years.

We settled down for a game. The first game was just about laying down stones. No pressure, we laid stones down just to get in the “groove” really. After our first foray, Doug and I played an instructional game. This for me was the most challenging and the most informative. When Doug teaches about the game, he doesn’t just show you what the possible good plays are–he tells you why. He tells you how a play affects the game in the “local” sense as well as it’s relationship to the overall dynamics of the board.

I have to say that it’s fascinating to learn how a stone played on one end of the board has “everything to do” with stones played on the other end.

The game is visceral, it’s fun. The “feeling” of your playing with but also cooperating with your opponent has a “flow” that’s easier experienced than described.

September 17, 2010 0

Ride Maps

By in Cycling, Information Technology

Ride Maps

Here’s a sampling of the ride maps created via GPS…

This topographic map features color coding by elevation:

It’s all about speed, after all. Here’s a route map color coded to pace:

Here’s an elevation profile:

September 17, 2010 0

Carbon Fiber: Accept No Substitutes

By in Cycling

I have a goal. I will purchase a carbon fiber bike this spring. I’ve read through the reviews (that need to be taken with a grain of salt) and reports from an independant bike frame tester.

The best candidate so far is the CUBE Agree GTC race.

[caption id="attachment_58" align="aligncenter" width="693" caption="Cube Agree GTC Race"]Cube Agree GTC Race[/caption]

I’m excited to hear that John would like to take our riding up a step and do some winter training. This is good as it compliments my winter activities. John himself is definitely a strong rider. We have a good mix of just relaxing with a little competitiveness. It would be tough to ask for better. Also, in the course of full disclosure, my seeing John glide up Council Crest on his carbon definitely played a role in my decision to buy.

I’ll ride my Olmo this winter and start fresh this spring with fresh carbon. I told Chris yesterday that if I achieve the goal of getting this bike I _will_ ride the Seattle To Portland this year.

Stay tuned–look for maps showing our rides on this site.

September 16, 2010 0

Sattelite Photos with Tagged Image Locations

By in Programming

Have a GPS and a digital camera and would like to have a map depicting where you took your images? No problem.

Here are two scripts I wrote (one in Shell, the other in Ruby) that grabs a satellite photo based on your GPS’s data and places a point that corresponds where you took a picture. This script synchronizes your track log and your images’ header information to pull up a photo showing where you took each image. Here’s a sample:

The scripts require GPS drive and geo-map.

Main Script

(more…)