Friday, July 10, 2015

Worf version 1.1 released

Worf version 1.1 has just been released.  This update changes the filtering from time domain convolution to frequency domain convolution.  The result is drastically improved performance and lower battery drain.

The filtering step has been moved to before the plotting step, so you can actually see the effects of your filter in real time prior to recording.

I also changed the plotting routine to highlight samples that are above threshold in green.  Here's a new screenshot:


I also learned that the ability to skip silence using the custom thresholds is actually called "SkipSilence" in other audio apps.  Although Worf is still unique in that you can draw your own threshold over the audio spectrum to specify exactly how you want things to be skipped.  Other apps seem to call it a "beta" function and apply their own fixed default level.


UPDATE: I decided to throw in a notification message while recording in case people accidentally leave the recording thread running in the background.  I also capped the maximum length of a single record file to 2 hours (about 110 MB) so people don't accidentally fill up their storage by leaving it running for days on end.  I had to roll the revision to 1.2 for these things.


Tuesday, June 16, 2015

Worf is Live!



Worf 1.0 is now live on Google Play and available for free.  The initial version contains all the advertised functionality and has been tested on multiple devices for stability and performance.


I will respond to feedback and any issues people may run in to.

Here's my running wish list for the next version 1.1:
- Re-write MP3 parser to better support music files for playback
- Allow the application of threshold and filters to audio playback
- Resolve whatever issues people discover in version 1.0

Let me know how it goes....


Monday, June 15, 2015

Android Apps - Creating Worf

For the past few months I have been learning the Android development environment and how to create apps for the Google Play store.  My first release is an app called Worf, an advanced audio recorder and spectrum analyzer.  Worf is a great little utility that demonstrates several important concepts of digital signal processing.

For starters, Worf listens to the device's microphone and plots a real-time frequency spectrum.  This alone is not particularly impressive given the number of free spectrum analyzer's already available for Android.  But it was a necessary first step nonetheless to get familiar with audio capture.  There are a number of methods and pre-built libraries floating around for plotting data and whatnot, but I chose to write my own plot routines using the raw Android SDK methods.  It really wasn't that hard and probably easier than trying to learn someone else's library.  I went through a number of different FFT algorithms in both Java and C before settling on a C++ version I really like.  It seems most modern hardware is powerful enough to run any of them in real time, so it's just a matter of personal preference.  I wanted to get familiar with running native C and C++ through JNI anyway, so this was a good excuse.  After performing the FFT and plotting the frequency domain, the data is carted off to a LAME MP3 converter algorithm to render an audio output file.

Next I added a threshold that the user can draw with their finger over top of the audio spectrum.  Amplitude spikes that rise above the threshold anywhere in the spectrum are flagged.  Only the FFT buffers that contain flagged data are actually written to file, the rest are discarded.  After playing around a bit I decided the optimal approach was to capture four sets of data at a time for FFT processing, perform the threshold check on the first and last sets, and then write all four to file if either of the two crossed the threshold.  This provides a little bit of a buffer around the captured audio and helps to prevent things from getting cut off.

Finally, I added a filter that is also drawn in a manner similar to the threshold, but is much much more complicated.  Initially I made the naive mistake of thinking I could filter in the frequency domain by attenuating the data according to the shape of the curve that the user had drawn.  Then an inverse-FFT would yield the filtered time domain data for recording to file.  Well that's not really a good idea.  It sort of works when you get complicated with overlapping FFT's, but the main issue is that the spectrum plot is actually a magnitude, and not the full complex representation of the signal.  Without the imaginary parts it seems the signal quality after attenuation is not very good.

The better method I eventually settled on was to use time-domain filtering.  I chose a Kaiser-Bessel window function (I suppose a variety of others would work as well) to create a band-pass FIR filter.  Then I string together a whole set of FIR filters in parallel centered on each point of the FFT and attenuating the signal by the amount drawn by the user in the filter curve.  Turns out you just need to sum the FIR coefficients, so it's pretty easy and not CPU intensive.  This seems to perform pretty well and doesn't degrade the audio quality (very much).

Depending on the level of interest in this app and these implementation notes, I may post some of the specific algorithms used along with more details in a series of follow up posts.

I am releasing Worf as a free app in the Google Play store later this week.  I just want to do a little more testing with it first to see if there are any hidden bugs remaining.  The link below should work once it is released.

Android app on Google Play
Download Worf from Google Play


Tuesday, May 10, 2011

Education in America - job salary by field of employment

One of the most sought after but least discussed topics in the workplace is exactly how much a particular position pays on average compared to other jobs in the same or different fields.   Colleges often advertise the value of an education in America in terms of how much more a college graduate can earn versus a high school graduate, but that doesn't really give any specific real world information. The news media loves to publish the occasional "top jobs" list but they are very high level and cursory and intended more as entertainment than reference information.  In previous articles on education in America, the percentage of college graduates by field was presented, as well as the unemployment rate by field.  Here we look at salary and the number of people employed by field.

The U.S. Bureau of Labor Statistics is the only real go-to source for information on jobs, employment, and salaries.  Unfortunately, pulling information out of their database to make meaningful comparisons to was a bit of a chore.  But here it is (click on plots to view full size):

Salary by Job Field, 2009 and 1999


Not surprisingly, management pays the most, followed by law, engineering and healthcare fields.

Number of employees by job field, 2009 and 1999


Evidently there are a ton of office administrative support people out there, but very few cops (protective services).  Finally, the most meaningful chart shows the trends in these professions over the last 10 years.

Salary and employment growth by job field, 1999 to 2009



Every profession saw salary increases from 1999 to 2009.  For reference, inflation over that same time period was 28%. So in reality, some of the jobs towards the right of the plot more or less broke even, while personal care and services actually lost money.  The raw number of managers has actually decreased over the last 10 years, however their salaries have increased faster than any other profession.  The negative employment percentages towards the right of the graph could be attributed to the recession and high unemployment rate in 2009.

Finally, the high level categories in the previous plots don't capture the whole picture.  There is quite a bit of variation in job types and salaries within each category.  Below is a detailed table highlighting the main jobs in each category.  I had to do some pruning and cleanup on the raw data out of the Bureau of Labor Statistics.  Only the main and/or easily recognizable jobs in each field are listed, so the totals for each field are more than the sum of the jobs listed below them.


20091999% Change
Title# EmployedAverage Salary# EmployedAverage Salary# EmployedAverage Salary
All Occupations130,647,61043,460
Management6,116,380102,9008,063,41064,740-24%59%
Chief executives297,640167,280597,060101,240-50%65%
General and operations managers1,689,680110,5502,305,61065,910-27%68%
Marketing managers169,330120,070202,71071,010-16%69%
Sales managers328,980111,570367,64069,560-11%60%
Financial managers495,180113,730646,05069,100-23%65%
Construction managers204,76093,290240,49060,160-15%55%
Engineering managers178,110122,810248,21081,560-28%51%
Business and Financial6,063,67065,9004,361,98046,10039%43%
Claims adjusters, examiners, and investigators273,93058,780154,77041,96077%40%
Business operations specialists1,036,45065,960
Accountants and auditors1,106,98067,430843,16044,32031%52%
Financial analysts235,24085,240142,82056,34065%51%
Personal financial advisors149,46094,18079,97064,68087%46%
Loan officers298,20063,210200,18045,21049%40%
Engineering5,716,42074,9405,126,46053,26512%41%
Computer software engineers880,70093,395496,63066,00577%41%
Network and computer systems administrators338,89070,930204,68050,09066%42%
Architects101,63078,88071,04053,41043%48%
Surveyors50,36057,42050,15036,4000%58%
Aerospace engineers70,57096,27071,79064,550-2%49%
Biomedical engineers14,76082,5506,45052,430129%57%
Chemical engineers29,00091,67028,63064,2501%43%
Civil engineers259,32081,180209,10055,66024%46%
Computer hardware engineers65,410101,41060,42066,9608%51%
Electrical engineers151,66086,250149,21061,5202%40%
Environmental engineers50,61080,75051,45057,050-2%42%
Industrial engineers209,30077,090155,91057,45034%34%
Mechanical engineers232,66080,580202,91057,01015%41%
Nuclear engineers16,710100,3509,58072,87074%38%
Petroleum engineers25,540119,9609,64072,780165%65%
Engineering technicians514,06049,496566,00038,593-9%28%
Science1,308,38065,660909,53045,66044%44%
Biochemists and biophysicists22,86088,55011,81056,17094%58%
Microbiologists16,26071,98015,63050,6904%42%
Zoologists and wildlife biologists17,46060,67011,12043,40057%40%
Biological scientists29,63069,430
Medical scientists101,76084,76021,20055,880380%52%
Life scientists12,32072,590
Astronomers1,240102,74068075,86082%35%
Physicists13,630111,25010,29076,14032%46%
Atmospheric and space scientists8,32085,1607,17053,58016%59%
Chemists79,91072,74073,84051,5808%41%
Environmental scientists83,53067,36053,61047,00056%43%
Geoscientists31,86092,71020,94061,00052%52%
Economists13,16096,32014,49061,550-9%56%
Market research analysts226,41067,50067,67052,680235%28%
Psychologists108,59078,26592,46049,72017%57%
Anthropologists and archeologists5,57057,2303,22037,01073%55%
Historians3,62056,3501,51041,350140%36%
Political scientists3,970101,0504,28072,860-7%39%
Social scientists29,25073,450
Biological technicians74,56041,14039,58032,06088%28%
Chemical technicians64,42043,90078,73036,080-18%22%
Social Services1,891,32042,7501,404,54031,64035%35%
Substance abuse and behavioral disorder counselors78,47040,42057,29028,56037%42%
Educational, vocational, and school counselors251,05055,030190,93041,49031%33%
Marriage and family therapists26,45049,02018,53035,66043%37%
Mental health counselors106,92041,71062,91029,43070%42%
Rehabilitation counselors112,69034,71093,13026,52021%31%
Child, family, and school social workers277,67043,540262,57031,7206%37%
Medical and public health social workers133,51048,340101,68035,40031%37%
Mental health and substance abuse social workers127,14041,35072,73031,15075%33%
Probation officers and correctional treatment specialists92,91050,50078,93038,03018%33%
Social and human service assistants344,05029,880242,53022,76042%31%
Clergy42,67046,96026,00032,21064%46%
Legal999,02095,820858,32066,78016%43%
Lawyers556,790129,020464,25090,36020%43%
Judges, magistrate judges, and magistrates26,350103,99023,15067,15014%55%
Paralegals and legal assistants246,81050,080175,87036,55040%37%
Education8,488,74049,5307,344,83036,04016%37%
Postsecondary teachers1,392,85073,387798,01052,42175%40%
Preschool teachers389,66027,450339,31019,61015%40%
Kindergarten teachers181,81050,380158,25036,77015%37%
Elementary school teachers1,544,30053,1501,357,34039,56014%34%
Middle school teachers665,42053,550570,01039,69017%35%
Secondary school teachers1,091,71055,150947,01041,43015%33%
Vocational education teachers92,98054,420107,33041,710-13%30%
Special education teachers477,31054,980410,41040,35716%36%
Adult literacy, remedial education, and GED teachers and instructors68,43050,39056,88032,80020%54%
Librarians150,52055,670137,76041,2709%35%
Teacher assistants1,275,41024,2801,115,82017,40014%40%
Arts, Entertainment, Sports, and Media1,745,67051,7201,551,60037,65013%37%
Art directors31,66091,52019,19061,68065%48%
Multi-media artists and animators28,80062,81030,53041,330-6%52%
Commercial and industrial designers29,17061,40038,35047,910-24%28%
Fashion designers15,78074,4109,60052,58064%42%
Floral designers51,47024,94059,41018,980-13%31%
Graphic designers200,87047,820119,82036,21068%32%
Interior designers46,01051,99029,69038,36055%36%
Merchandise displayers and window trimmers61,28028,48051,53021,56019%32%
Actors39,880*84,79050,620-53%
Producers and directors79,78086,87039,20047,230104%84%
Athletes and sports competitors13,62080,95010,62069,44028%17%
Coaches and scouts179,83035,74065,82032,010173%12%
Umpires, referees, and other sports officials14,86028,4908,15023,51082%21%
Dancers10,700*14,91024,750-28%
Choreographers14,70042,13011,17032,37032%30%
Music directors and composers14,33053,4106,31034,750127%54%
Musicians and singers47,260*46,44037,5102%
Radio and television announcers42,41038,86050,41025,640-16%52%
Reporters, correspondents, analysts51,95055,63064,59035,270-20%58%
Public relations specialists242,67059,370118,28040,780105%46%
Editors105,04058,44095,21042,03010%39%
Technical writers46,27065,61046,68046,940-1%40%
Writers and authors43,39064,56045,67045,500-5%42%
Audio and video equipment technicians46,07042,45039,09034,02018%25%
Sound engineering technicians15,56053,9409,38035,09066%54%
Photographers57,76036,37066,07025,710-13%41%
Camera operators, television, video, and motion picture17,54049,59017,33031,7301%56%
Film and video editors17,55063,68012,77044,38037%43%
Healthcare Practitioners7,200,95069,6906,001,95045,25020%54%
Chiropractors26,31080,39011,54070,930128%13%
Dentists86,270156,85069,360106,13024%48%
Orthodontists5,410206,190
Dietitians and nutritionists53,22053,23041,32037,35029%43%
Optometrists26,480106,96021,40077,75024%38%
Pharmacists267,860106,630226,30063,03018%69%
Anesthesiologists37,450211,75025,910123,78045%71%
Family and general practitioners99,000168,550134,490104,090-26%62%
Internists, general48,270183,99048,740123,280-1%49%
Obstetricians and gynecologists20,380204,47018,780135,4309%51%
Pediatricians, general29,460161,41018,940112,76056%43%
Psychiatrists22,210163,66017,870103,66024%58%
Surgeons44,560219,77048,450135,660-8%62%
Physician assistants76,90084,83056,75050,65036%67%
Podiatrists9,720131,7304,470100,090117%32%
Registered nurses2,583,77066,5302,205,43044,47017%50%
Occupational therapists97,84070,68078,95051,91024%36%
Physical therapists174,49076,220131,05058,35033%31%
Veterinarians54,13090,11039,25066,06038%36%
Medical and clinical laboratory technologists166,86055,620145,75039,31014%41%
Dental hygienists173,90067,86090,05048,15093%41%
Cardiovascular technologists and technicians48,07049,73041,49033,28016%49%
Diagnostic medical sonographers51,63063,64029,28043,76076%45%
Nuclear medicine technologists21,67068,45017,88042,43021%61%
Emergency medical technicians and paramedics217,92033,020172,36023,28026%42%
Dietetic technicians24,51028,53029,19021,000-16%36%
Pharmacy technicians331,89028,940196,43020,05069%44%
Veterinary technologists and technicians79,20030,58047,47021,43067%43%
Opticians, dispensing60,84034,79058,86025,1903%38%
Athletic trainers15,26044,02016,67033,650-8%31%
Healthcare Support3,886,69026,7102,970,78019,78031%35%
Home health aides955,22021,620577,53018,81065%15%
Nursing aides, orderlies, and attendants1,438,01024,9801,308,74017,86010%40%
Psychiatric aides62,61027,43051,10022,39023%23%
Massage therapists55,92039,78021,91028,740155%38%
Dental assistants294,02034,000175,16024,13068%41%
Medical assistants495,97029,450281,48022,65076%30%
Protective Services3,172,42041,7402,958,73029,6507%41%
Fire fighters305,50047,270252,73034,07021%39%
Correctional officers and jailers455,35042,610381,25031,07019%37%
Detectives and criminal investigators110,38065,86083,34047,62032%38%
Fish and game wardens7,53054,9508,22041,940-8%31%
Police and sheriff's patrol officers641,59055,180581,86038,71010%43%
Animal control workers15,32033,5608,30023,85085%41%
Private detectives and investigators31,25047,13030,69030,1202%56%
Security guards1,028,83026,4301,088,47018,610-5%42%
Crossing guards68,47025,43068,31017,8700%42%
Lifeguards, ski patrol, and other recreational protective service workers115,64020,490
Food Preparation and Serving11,218,26020,8809,687,97015,60016%34%
Chefs and head cooks94,30044,240118,07028,040-20%58%
Cooks, fast food539,52018,230418,40013,61029%34%
Cooks, institution and cafeteria383,54023,870438,66017,420-13%37%
Cooks, restaurant898,82023,110656,54017,73037%30%
Food preparation workers849,40020,420878,65015,740-3%30%
Bartenders492,48020,970358,45014,70037%43%
Waiters and waitresses2,302,07020,3802,039,95013,43013%52%
Dishwashers512,99018,330538,36014,090-5%30%
Cleaning and Maintenance4,269,48024,9704,274,20018,9100%32%
Janitors2,090,40024,1202,090,56018,2200%32%
Maids and housekeeping cleaners887,89020,840913,47015,530-3%34%
Pest control workers63,50032,10040,24024,12058%33%
Landscaping and groundskeeping workers859,96025,340739,46019,38016%31%
Tree trimmers and pruners37,83032,09047,89023,770-21%35%
Personal Care and Service3,461,91024,6802,556,92020,30035%22%
Gaming dealers86,90020,29087,39014,120-1%44%
Ushers, lobby attendants, and ticket takers104,36019,61088,59014,05018%40%
Amusement and recreation attendants257,35019,450190,60014,92035%30%
Barbers10,55027,65014,15020,970-25%32%
Hairdressers, hairstylists, and cosmetologists349,21027,070314,75020,80011%30%
Baggage porters and bellhops49,38023,58059,58017,110-17%38%
Concierges20,47028,76016,44022,42025%28%
Tour guides and escorts31,63025,99035,78020,100-12%29%
Flight attendants95,81043,350123,31047,910-22%-10%
Child care workers595,65020,940377,11015,43058%36%
Personal and home care aides630,74020,280300,50016,060110%26%
Fitness trainers and aerobics instructors228,17035,340127,31027,30079%29%
Sales13,715,05036,02012,938,13027,0606%33%
Cashiers3,439,38019,0303,162,09015,2909%24%
Counter and rental clerks416,95024,680392,56016,6906%48%
Parts salespersons208,35030,460265,38025,410-21%20%
Retail salespersons4,209,50024,6303,729,04019,21013%28%
Advertising sales agents152,42053,190142,83041,4007%28%
Insurance sales agents325,71061,330241,73047,69035%29%
Travel agents76,99032,450111,13024,660-31%32%
Demonstrators and product promoters80,91026,64095,16021,420-15%24%
Models1,51036,4205,22021,110-71%73%
Real estate brokers48,38078,36026,76062,19081%26%
Real estate sales agents151,55053,100107,68036,99041%44%
Telemarketers307,73024,760485,65021,100-37%17%
Door-to-door sales workers8,46026,32036,13028,940-77%-9%
Office and Administrative Support22,336,45032,99022,562,48025,310-1%30%
Bill and account collectors403,10032,560383,09024,8605%31%
Billing and posting clerks and machine operators493,78032,900551,41023,880-10%38%
Bookkeeping, accounting, and auditing clerks1,757,87034,7501,619,87025,2509%38%
Tellers576,58024,780453,14018,33027%35%
Customer service representatives2,195,86032,4101,789,62025,36023%28%
Receptionists and information clerks1,052,12026,010987,68019,8707%31%
Postal service mail carriers339,03048,940352,55036,610-4%34%
Shipping, receiving, and traffic clerks715,13029,840886,23022,080-19%35%
Stock clerks and order fillers1,864,41023,4601,800,84019,6504%19%
Executive secretaries and administrative assistants1,361,17044,0101,316,29030,8703%43%
Secretaries1,797,67031,0601,582,08024,13014%29%
Farming419,20023,990463,36018,000-10%33%
Agricultural inspectors14,03041,86010,52030,08033%39%
Graders and sorters, agricultural products40,16020,64062,38015,520-36%33%
Agricultural equipment operators22,42025,22017,63017,64027%43%
Farmworkers and laborers233,65019,780215,08014,1509%40%
Fishers and related fishing workers67026,600
Forest and conservation workers5,84029,41011,78020,120-50%46%
Logging equipment operators23,63032,87033,23025,310-29%30%
Log graders and scalers2,94035,5505,50026,180-47%36%
Construction and Extraction5,751,63043,3505,938,86033,650-3%29%
Brickmasons and blockmasons87,78049,25098,53041,380-11%19%
Carpenters743,76043,640771,03034,420-4%27%
Cement masons and concrete finishers165,70039,410151,76031,2109%26%
Construction laborers856,44033,190763,45026,51012%25%
Construction equipment operators368,20044,180324,35034,76014%27%
Drywall and ceiling tile installers102,88041,080118,30034,090-13%21%
Electricians579,15050,850611,92042,180-5%21%
Painters, construction and maintenance214,24037,320260,88029,280-18%27%
Plumbers, pipefitters, and steamfitters400,97049,870413,17038,750-3%29%
Roofers108,18037,390115,28029,870-6%25%
Sheet metal workers146,69044,890231,69033,110-37%36%
Construction and building inspectors90,73053,55067,01039,74035%35%
Highway maintenance workers139,49035,310139,54026,7300%32%
Roustabouts, oil and gas61,32034,19029,86023,260105%47%
Installation, Maintenance, and Repair5,114,15042,2105,140,21032,810-1%29%
Telecommunications equipment installers189,85052,990172,70041,13010%29%
Security and fire alarm systems installers63,69039,83038,35030,81066%29%
Aircraft mechanics and service technicians112,13052,970125,97039,280-11%35%
Automotive service technicians and mechanics606,99037,880587,32030,1303%26%
Bus and truck mechanics and diesel engine specialists232,81041,590273,32031,800-15%31%
Heating and air conditioning mechanics244,41043,670187,85032,04030%36%
Industrial machinery mechanics276,23046,160176,07036,21057%27%
Maintenance and repair workers, general1,268,93036,5201,201,69026,9306%36%
Telecommunications line installers and repairers162,40048,310158,99035,7902%35%
Production8,927,13033,29012,620,92025,400-29%31%
Electrical and electronic equipment assemblers193,57030,690387,43021,840-50%41%
Team assemblers997,39028,8401,302,82022,200-23%30%
Assemblers and fabricators, all other267,78032,280
Bakers140,51025,350176,08019,990-20%27%
Butchers and meat cutters125,51030,270138,87024,890-10%22%
Meat, poultry, and fish cutters and trimmers168,70022,900159,89017,3706%32%
Cutting, punching, and press machine setters, operators, and tenders, metal and plastic209,73030,480353,30023,640-41%29%
Machinists380,72038,940419,80030,540-9%28%
Welders, cutters, solderers, and brazers357,74036,630410,04027,870-13%31%
Laundry and dry-cleaning workers211,49020,790217,35015,760-3%32%
Power plant operators36,86060,27034,31043,1107%40%
Chemical plant and system operators45,75054,01064,32039,330-29%37%
Gas plant operators14,04055,86016,74043,080-16%30%
Inspectors, testers, sorters, samplers, and weighers430,45034,840577,65027,140-25%28%
Jewelers and precious stone and metal workers23,41036,62028,69026,360-18%39%
Packaging and filling machine operators and tenders338,92027,320379,76020,790-11%31%
Transportation8,844,70032,1809,538,82024,630-7%31%
Airline pilots, copilots, and flight engineers74,420117,06088,04098,280-15%19%
Commercial pilots29,18073,06018,78056,24055%30%
Air traffic controllers24,420106,99022,62073,1908%46%
Bus drivers, transit and intercity177,51035,990160,21026,45011%36%
Bus drivers, school459,48028,050463,86020,460-1%37%
Driver/sales workers363,05026,690385,21022,520-6%19%
Truck drivers, heavy and tractor-trailer1,550,93039,2601,558,40031,9000%23%
Truck drivers, light or delivery services834,78031,1201,085,05023,530-23%32%
Taxi drivers and chauffeurs167,74023,930119,63018,20040%31%
Locomotive engineers43,56053,59019,94048,050118%12%
Subway and streetcar operators6,05052,800
Sailors and marine oilers31,95037,31027,20025,82017%45%
Captains, mates, and pilots of water vessels30,45070,74020,66041,46047%71%
Parking lot attendants129,99020,600109,34015,35019%34%
Service station attendants79,48020,820109,05015,770-27%32%
Crane and tower operators40,77047,70053,83033,710-24%42%
Excavating and loading machine and dragline operators57,99038,54062,36031,460-7%23%
Industrial truck and tractor operators568,27031,240590,71025,650-4%22%
Cleaners of vehicles and equipment298,50022,110302,38016,650-1%33%
Freight, stock, and material movers2,135,79025,2902,035,64019,7505%28%
Packers and packagers, hand706,24021,7801,114,33016,280-37%34%
Refuse and recyclable material collectors128,94033,760135,32025,020-5%35%



References:

http://www.bls.gov