Dev and Admin Tools

From awctech wiki
Jump to: navigation, search

Git Basics

If created new repo in web UI and want to add new project

  • git init create a new git local repo in current directory
  • git remote add origin [remote repository URL] sets the new remote repository
  • git remote -v verifies the new remote URL
  • git push origin master push local repo changes to remote repo you specified as the origin

File editing

  • Working_Dir --add--> Index --commit--> Head --push--> Source_Repo
  • git pull pulls changes from remote repo and merges any pulled commits into your current local branch, also consider git fetch
  • git add file.txt stages edited file to the local git Index
  • git checkout -- file.txt revert local change made in working directory (does not apply to Index) to what is in Head:
  • git rm file.txt removes file from working directory and local git Index
  • git status shows contents of local git Index
  • git commit -m "Commit Message" commits all staged changes to the local git Head
  • git push origin master pushes all commits to the source repo

Example workflow for editing another user's repo

  • Fork other_user's repo in UI and clone locally: local repo <----> remote forked repo (origin) <----> other_user's repo (upstream)
  • git remote add upstream [other_user's repo url] create upstream remote to original repo (note that remotes point to loc of repos, not branches)
  • git fetch upstream pull a cache of upstream
  • git checkout -b FEATURE_X create a new branch named "FEATURE_X" and switch to it
  • git push origin FEATURE_X update locally and push to remote forked repo the local branch FEATURE_X
  • Create pull request in UI to upstream (other_user/master), assume accepted/merged
  • git checkout master switch back to local master
  • git pull --rebase upstream master update local master with other_user's master
  • git push origin master push local master to remote repository


Basics of Branching/Merging:

Simple guide:

Check the git config: git config --list

Check who made edits on each line of a file: git blame file.txt

Pip Workflow

This pip workflow gives you:

  • List of packages added by the developer, with some package versions frozen, in the standard requirements.txt
  • List of all package versions, including dependencies, separate from the packages added by the developer

In the requirements.txt, put the first line as ## Packages added by developer followed by all pip packages you add:

$ cat requirements.txt
## Packages added by developer

Run the following when you want to update documentation for package versions, separated by what developer added:

$ pip freeze -r requirements.txt > pip-freeze-req.txt
$ cat pip-freeze-req.txt
## Packages added by developer
## The following requirements were added by pip freeze:

Edit requirements.txt if you want to freeze the versions of any packages:

$ cat requirements.txt
## Packages added by developer

When you deploy your project, pip install from requirements.txt as normal

$ pip install -r requirements.txt

Tmux basics

  • From bash: tmux to start session, tmux attach-session -d to reload session
  • Within tmux press ctrl-b then [key combos] to issue a tmux command
  • % " [arrow_keys] to split and navigate screns. Hold down ctrl-b with [arrow_keys] to resize
  • z goes to and returns from full screen
  • c for new desktop, [comma] to rename, [num_keys] to navigate desktops
  • [enter tilde period] key combo to disconnect ssh session and keep tmux layout, bash exit will loose tmux layout
  • [ goes into scroll mode, esc to exit
  • { moves current window around other panes


Unicode is way of representing each character, across all world languages, in a Unicode code point (Unicode is not an encoding).

Dog in Unicode is: U+0044 U+006F U+0067

Note that examples are done using Python 2.7 where the default 'str' type is byte string, in Python 3 the default 'str' type is unicode.

>>> u_text = u'Café ZerФ'
>>> u_text
u'Caf\\xe9 Zer\\u0424'
>>> type(u_text)
<type 'unicode'>

The special characters in u_text ending each word are U+00E9 and U+0424. Because U+00E9 can be represented in one byte, python displays it as \\xe9. Whether the escape character starts with \\x (for hex) or \\u (for Unicode) they are both Unicode code points in hexidecimal. The u preceding the single quote in python denotes it is Unicode.

UTF-8 encoding is one very common way of storing any of the Unicode code points in memory. For code points 0-127 UTF-8 stores using 8-bit bytes, for 128 and above UTF-8 stores using multiple bytes (up to 6 bytes). UTF-8 is a variable-length encoding.

>>> s_text = u_text.encode('utf-8')
>>> s_text
'Caf\\xc3\\xa9 Zer\\xd0\\xa4'
>>> type(s_text)
<type 'str'>

s_text was encoded into a UTF-8 byte string (<type 'str'> = byte string). Each character in Caf and Zer has its code point stored in memory as an 8-bit byte (which is how ASCII stores them). Each special character is stored in memory using 2 bytes, denoted by \\xc3\\xa9 and \\xd0\\xa4

>>> sys.getdefaultencoding()

This Python 2.7 environment states its default encoding as ASCII. Note that ASCII encoding is a way of storing characters in memory, it supports only 128 characters. ASCII maps 1:1 onto UTF-8, but not vice versa.

>>> default1 = 'sun'
>>> default1.decode('ascii')
>>> type(default1)
<type 'str'>

The default1 variable is created as type byte string, it is not Unicode and therefore not decoded. The encoding is ASCII - it can be decoded to Unicode by using the ASCII codec.

>>> default2 = 'Café'
>>> default2
>>> default2.decode('ascii')
Traceback ... UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 ...

The default2 variable is also created as type byte string, it is not Unicode and therefore not decoded. The encoding is not ASCII, python gives an error when trying to decode into Unicode using the ASCII codec since ASCII does not contain the accented 'e'.

>>> default2.decode('utf-8')
>>> default2.decode('latin1')
>>> print default2.decode('latin1')
>>> print default2.decode('utf-8')

The default2 variable successfully decodes into Unicode using either the Latin1 or UTF-8 encoding. However only the UTF-8 codec does it correctly, which suggests python encoded this as UTF-8 when it found a letter outside the 128 ASCII characters.

"When strings contain non-ASCII characters, they can either be 8-bit strings (encoded strings), or they can be Unicode strings (decoded strings)." -

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

What every programmer absolutely, positively needs to know about encodings and character sets to work with text

Apache notes

  • VirtualHost block in httpd.conf allows for server config specific to a subdomain (or even separate web sites / IPs) -
  • Directory block in httpd.conf allows for server config specific to a directory -
  • .htaccess files are distributed (sitting alongside .html files) configuration files and perform the same task as the Directory block in httpd.conf, but slower since they are loaded on each page request. Should only be used when there is no access to the Apache server.

Selenium Python

API to write automated Selenium WebDriver browser tests with Python

Install pip:

Full Selenium Python documentation:

Saucelabs can allow Selenium scripts to run on their wide variety OS/mobile/browser setups:


In Linux, switch to postgres user and go into psql command line (more basics and tutorial)

$ sudo -u postgres psql

A few commands inside the psql command line:
# \l List databases
# \c database-name Connect to database
# \dt List tables in current database
# SELECT * FROM TABLE1; Run SQL statements by entering them individually or in succession and terminate with semicolon to execute query
# \q Quit psql

Database dump

 $ pg_dump dbname > /home/user/db-dump.sql 

View Queries via Log

See queries in real time by changing log level:


Mark error script performed

"Task that indicates that the failed script was manually performed. The script will NOT be run again in the next update. No scripts will be executed by this task."

If on a database update a script runs and partially executes database changes before having an error: You can fix script, manually execute the rest of the sql in the script, and tell dbmaintain the script has been performed

$ ./ markErrorScriptPerformed scripts/01_v1.0/01_products_and_orders.sql 

Mark the database as up-to-date

"This operation updates the state of the database to indicate that all scripts have been executed, without actually executing them. This can be useful when you want to start using DbMaintain on an existing database, or after having fixed a problem directly on the database."

In the dbmaintain_scripts table it updates the executed_at column to the current datetime for all scripts (undesirable).

Can be used if there is "irregular script update" exception, but better to fix the script (change name, update md5 checksum in database if same content, etc), and then update database.

Can be used if the database scripts were renamed after an update and dbmaintain returns "Renamed indexed script", but may be better to mark error script performed.

Mark error script reverted

"Task that indicates that the failed script was manually reverted. The script will be run again in the next update. No scripts will be executed by this task."

Also you can fix script, manually rollback changes, and use markErrorScriptReverted to tell dbmaintain the script can be re-run on database update.

See more DbMaintain operations: path/to/scriptFolderOrArchive


Helpful Links

General Links