Data Munging

Sum a list of numbers

cat list | paste -s -d+ | bc

Remove all but one trailing newlines

sed -e :a -e '/^\n*$/{$d;N;};/\n$/ba' file

Make sure a file has a trailing newline

tail -c1 file | read -r _ || echo >> file

Double-space a file

sed -G

Add a newline after every two lines

sed '0~2 s/$/\n/'

Replace newlines with arbitrary text

awk '{ print }' RS='\n' ORS='arbitrary_text'

Replace newlines with commas

paste -s -d ,

Work with human-readable e.g. disk ^2 sizes

numfmt --from=iec and numfmt --to=iec

Search Oracle lobs for strings

select whatever from atable where (dbms_lob.instr(lob_column,utl_raw.cast_to_raw('text_to_look_for')) > 0)

Quick n dirty CSV awk

BEGIN {
    FPAT = "([^,]+)|(\"[^\"]+\")"
}

(doesn't handle newlines in quotes or escaped quotes though)

Paginated REST results

Paginated json coming back from a rest api? Use a python generator to return it.

Convert Unix time to a different TZ in python

import time
from datetime import datetime
from pytz import timezone, utc
london = timezone("Europe/London")
now = time.time()
t = datetime.utcfromtimestamp(now)
lt = utc.localize(t).astimezone(london)
print(lt.strftime('%Y-%m-%d %H:%M:%S %Z'))

Previous calendar month maths in python

import calendar
from datetime import datetime, timedelta

def prev_month_name():
    today = datetime.today().date()
    first_day_this_month = today.replace(day=1)
    last_day_prev_month = first_day_this_month - timedelta(days=1)
    return calendar.month_name[last_day_prev_month.month]

def prev_month_year():
    today = datetime.today().date()
    first_day_this_month = today.replace(day=1)
    last_day_prev_month = first_day_this_month - timedelta(days=1)
    return last_day_prev_month.year

def prev_month_start_epoch():
    today = datetime.today().date()
    first_day_this_month = today.replace(day=1)
    last_day_prev_month = first_day_this_month - timedelta(days=1)
    first_day_prev_month = last_day_prev_month.replace(day=1)
    first_day_prev_month_zero_seconds = datetime(
        first_day_prev_month.year,
        first_day_prev_month.month,
        first_day_prev_month.day,
        0,
        0,
        0,
    )
    return (first_day_prev_month_zero_seconds - datetime(1970, 1, 1)).total_seconds()

def prev_month_end_epoch():
    today = datetime.today().date()
    first_day_this_month = today.replace(day=1)
    last_day_prev_month = first_day_this_month - timedelta(days=1)
    last_day_prev_month_midnightish = datetime(
        last_day_prev_month.year,
        last_day_prev_month.month,
        last_day_prev_month.day,
        23,
        59,
        59,
    )
    return (last_day_prev_month_midnightish - datetime(1970, 1, 1)).total_seconds() + 1