Skip to content

Commit

Permalink
Initial Setup
Browse files Browse the repository at this point in the history
  • Loading branch information
@rsdoiel
rsdoiel committed Feb 6, 2017
0 parents commit a0b48c4
Show file tree
Hide file tree
Showing 37 changed files with 2,804 additions and 0 deletions.
34 changes: 34 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so

# Folders
_obj
_test

# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out

*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*

_testmain.go

*.exe
*.test
*.prof

#
# Project specific files
#
*~
*.swp
*.zip
dist/*
bin/*

80 changes: 80 additions & 0 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@

# Installation

## Compiled version

*datatools* is a collection of command line programs run from a shell like Bash (or Powershell in Windows). If you download the repository a compiled version is in the dist directory. The compiled VERSION_NO matching your computer type and operating system can be copied to a bin directory in your PATH.

Compiled versions are available for Mac OS X (amd64 processor), Linux (amd64), Windows (amd64) and Rapsberry Pi (both ARM6 and ARM7)

### Mac OS X

1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
2. Open a finder window, find and unzip **datatools-VERSION_NO-release.zip**
3. Look in the unziped folder and find *dist/macosx-amd64/*
4. Drag (or copy) *jsoncols*, *jsonrange*, etc. to a "bin" directory in your path
5. Open and "Terminal" and run `jsoncols -h` to confirm you were successful

### Windows

1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
2. Open the file manager find and unzip **datatools-VERSION_NO-release.zip**
3. Look in the unziped folder and find *dist/windows-amd64/*
4. Drag (or copy) *jsoncols.exe*, *jsonrange.exe*, etc. to a "bin" directory in your path
5. Open Bash and and run `jsoncols -h` to confirm you were successful

### Linux

1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
2. Find and unzip **datatools-VERSION_NO-release.zip**
3. In the unziped directory and find for *dist/linux-amd64/*
4. Copy *jsoncols*, *jsonrange*, etc. to a "bin" directory (e.g. cp ~/Downloads/datatools-VERSION_NO-release/dist/linux-amd64/\* ~/bin/)
5. From the shell prompt run `jsoncols -h` to confirm you were successful

### Raspberry Pi

If you are using a Raspberry Pi 2 or later use the ARM7 VERSION_NO, ARM6 is only for the first generaiton Raspberry Pi.

1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
2. Find and unzip **datatools-VERSION_NO-release.zip**
3. In the unziped directory and find for *dist/raspberrypi-arm7/*
4. Copy *jsoncols*, *jsonrange*, etc. to a "bin" directory (e.g. cp ~/Downloads/datatools-VERSION_NO-release/dist/raspberrypi-arm7/\* ~/bin/)
+ if you are using an original Raspberry Pi you should copy the ARM6 version instead
5. From the shell prompt run `jsoncols -h` to confirm you were successful


## Compiling from source

If you have go v1.7.4 or better installed then should be able to "go get" to install all the **datatools** utilities and

```
go get -u github.com/caltechlibrary/datatools/...
```

Or for Windows 10 Powershell (assumes the Windows versions of Go and Git are previously installed)


```powershell
$Env:GOPATH = "$HOME"
go get -u github.com/caltechlibrary/datatools/...
```

or to install from source on Linux/Unix systems

```bash
git clone https://github.com/caltechlibrary/datatools src/github.com/caltechlibrary/datatools
cd src/github.com/caltechlibrary/datatools
make
make test
make install
```

Or for Windows 10 Powershell

```powershell
$Env:GOBIN = "$HOME\bin"
git clone https://github.com/caltechlibrary/datatools src/github.com/caltechlibrary/datatools
cd src\github.com\caltechlibrary\datatools
go install cmds\jsoncols\filefile.go
```

13 changes: 13 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@

Copyright (c) 2017, Caltech
All rights not granted herein are expressly reserved by Caltech.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
135 changes: 135 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
#
# Simple Makefile
#
PROJECT = datatools

VERSION = $(shell grep -m1 'Version = ' $(PROJECT).go | cut -d\" -f 2)

BRANCH = $(shell git branch | grep '* ' | cut -d\ -f 2)

build: bin/csvcols bin/csvjoin bin/jsoncols bin/jsonrange bin/xlsx2json bin/xlsx2csv bin/csv2mdtable bin/csv2xlsx

bin/csvcols: datatools.go cmds/csvcols/csvcols.go
go build -o bin/csvcols cmds/csvcols/csvcols.go

bin/csvjoin: datatools.go cmds/csvjoin/csvjoin.go
go build -o bin/csvjoin cmds/csvjoin/csvjoin.go

bin/jsoncols: datatools.go cmds/jsoncols/jsoncols.go
go build -o bin/jsoncols cmds/jsoncols/jsoncols.go

bin/jsonrange: datatools.go cmds/jsonrange/jsonrange.go
go build -o bin/jsonrange cmds/jsonrange/jsonrange.go

bin/xlsx2json: datatools.go cmds/xlsx2json/xlsx2json.go
go build -o bin/xlsx2json cmds/xlsx2json/xlsx2json.go

bin/xlsx2csv: datatools.go cmds/xlsx2csv/xlsx2csv.go
go build -o bin/xlsx2csv cmds/xlsx2csv/xlsx2csv.go

bin/csv2mdtable: datatools.go cmds/csv2mdtable/csv2mdtable.go
go build -o bin/csv2mdtable cmds/csv2mdtable/csv2mdtable.go

bin/csv2xlsx: datatools.go cmds/csv2xlsx/csv2xlsx.go
go build -o bin/csv2xlsx cmds/csv2xlsx/csv2xlsx.go

website:
./mk-website.bash

status:
git status

save:
git commit -am "Quick Save"
git push origin $(BRANCH)

refresh:
git fetch origin
git pull origin $(BRANCH)

publish:
./mk-website.bash
./publish.bash

clean:
if [ -f index.html ]; then /bin/rm *.html;fi
if [ -d bin ]; then /bin/rm -fR bin; fi
if [ -d dist ]; then /bin/rm -fR dist; fi
if [ -f $(PROJECT)-$(VERSION)-release.zip ]; then rm -f $(PROJECT)-$(VERSION)-release.zip; fi

install:
env GOBIN=$(HOME)/bin go install cmds/csvcols/csvcols.go
env GOBIN=$(HOME)/bin go install cmds/csvjoin/csvjoin.go
env GOBIN=$(HOME)/bin go install cmds/jsoncols/jsoncols.go
env GOBIN=$(HOME)/bin go install cmds/jsonrange/jsonrange.go
env GOBIN=$(HOME)/bin go install cmds/xlsx2json/xlsx2json.go
env GOBIN=$(HOME)/bin go install cmds/xlsx2csv/xlsx2csv.go
env GOBIN=$(HOME)/bin go install cmds/csv2mdtable/csv2mdtable.go
env GOBIN=$(HOME)/bin go install cmds/csv2xlsx/csv2xlsx.go

dist/linux-amd64:
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csvcols cmds/csvcols/csvcols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csvjoin cmds/csvjoin/csvjoin.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/jsoncols cmds/jsoncols/jsoncols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/jsonrange cmds/jsonrange/jsonrange.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/xlsx2json cmds/xlsx2json/xlsx2json.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csv2xlsx cmds/csv2xlsx/csv2xlsx.go

dist/macosx-amd64:
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csvcols cmds/csvcols/csvcols.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csvjoin cmds/csvjoin/csvjoin.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/jsoncols cmds/jsoncols/jsoncols.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/jsonrange cmds/jsonrange/jsonrange.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/xlsx2json cmds/xlsx2json/xlsx2json.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csv2xlsx cmds/csv2xlsx/csv2xlsx.go

dist/windows-amd64:
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csvcols.exe cmds/csvcols/csvcols.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csvjoin.exe cmds/csvjoin/csvjoin.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/jsoncols.exe cmds/jsoncols/jsoncols.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/jsonrange.exe cmds/jsonrange/jsonrange.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/xlsx2json.exe cmds/xlsx2json/xlsx2json.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/xlsx2csv.exe cmds/xlsx2csv/xlsx2csv.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csv2mdtable.exe cmds/csv2mdtable/csv2mdtable.go
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csv2xlsx.exe cmds/csv2xlsx/csv2xlsx.go

dist/raspbian-arm7:
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csvcols cmds/csvcols/csvcols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csvjoin cmds/csvjoin/csvjoin.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/jsoncols cmds/jsoncols/jsoncols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/jsonrange cmds/jsonrange/jsonrange.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/xlsx2json cmds/xlsx2json/xlsx2json.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csv2xlsx cmds/csv2xlsx/csv2xlsx.go

dist/raspbian-arm6:
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csvcols cmds/csvcols/csvcols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csvjoin cmds/csvjoin/csvjoin.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/jsoncols cmds/jsoncols/jsoncols.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/jsonrange cmds/jsonrange/jsonrange.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/xlsx2json cmds/xlsx2json/xlsx2json.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csv2xlsx cmds/csv2xlsx/csv2xlsx.go


release: dist/linux-amd64 dist/macosx-amd64 dist/windows-amd64 dist/raspbian-arm7 dist/raspbian-arm6
mkdir -p dist
cp -v README.md dist/
cp -v LICENSE dist/
cp -v INSTALL.md dist/
cp -v csvcols.md dist/
cp -v csvjoin.md dist/
cp -v jsoncols.md dist/
cp -v jsonrange.md dist/
cp -v xlsx2json.md dist/
cp -v xlsx2csv.md dist/
cp -v csv2mdtable.md dist/
cp -v csv2xlsx.md dist/
zip -r $(PROJECT)-$(VERSION)-release.zip dist/

31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@

# datatools

Various utilities for simplifying JSON, Excel Workbook and CSV data work on the command line.

+ [csvcols](csvcols.html) - a tool for formatting command line arguments intoa CSV row or filtering CSV rows for specific columns
+ [csvjoin](csvjoin.html) - a tool to join to CSV files on common values in designated columns, writes combined CSV rows to stdout
+ [csv2mdtable](csv2mdtable.html) - a tool to render CSV as a Github Flavored Markdown table
+ [jsoncols](jsoncols.html) - a tool for exploring and extracting JSON values into columns
+ [jsonrange](jsonrange.html) - a tool for iterating for JSON maps and arrays
+ [xlsx2json](xlsx2json.html) - a tool for converting Excel Workbooks to JSON files
+ [xlsx2csv](xlsx2csv.html) - a tool for converting Excel Workbooks sheets to a CSV file(s)

Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
Windows 10 (amd64) and Raspbian (ARM6, ARM7). See https://github.com/caltechlibrary/datatools.

Use the utilities try "-help" option for a full list of options.


## Installation

_datatools_ is go get-able.

```
go get github.com/caltechlibrary/datatools/...
```

Or see [INSTALL.md](install.html) for details for installing
compiled versions of the programs.


12 changes: 12 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@

# Someday, Maybe

+ csv2json would convert CSV content to a 2d-JSON array
+ E.g. `cat file1.csv | csv2json`
+ csv2xlsx would take CSV content piped from stdin and write it do a sheet in an Excel file (creating the Excel workbook if needed)
+ E.g. `cat file1.csv | csv2xlsx MyWorkbook.xlsx "Sheet from file1.csv"`
+ csvfind would accept CSV input from stdin and output rows with matching column values
+ E.g. `cat file1.csv | csvfind -column=3 "Book"`
+ json2csv would convert a 2d JSON array to CSV output, it would comvert a JSON object/map to a column of keys next to a column of values
+ E.g. `cat data.json | json2csv`

Binary file added assets/liblogo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/orangebooks-logo-icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/orangebooks-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit a0b48c4

Please sign in to comment.