This implements the "git proj" sub-command, for managing large binary files, and repositories on external drives.
git proj [-v] [-V] [--version] [-h] [-H pStyle]
git proj [pSubCmd] [pSubCmdOpt] [pComOpt]
pSubCmd: init, remote, clone, push, pull, status
pSubCmdOpt: Just run: "git proj [pSubCmd] -H usage"
[common-options]: [-h] [-H pStyle] [-q -v -V N] [-x... -X N]
The "git proj" sub-command supports using git repo for versioning files to locally mounted external drives.
git-proj also mostly solves the issue with large binary files causing git repo "bloat". Large files are not versioned, they are only copied. So if a version important, just rename the file. (symlinks are used and versioned to point to the large binary files.)
This is a much simpler implementation than git-lfs for managing the versioning of large binary files. Since the large files are not versioned, i.e. only the last version is saved, this is a compromise on having "perfect" version control of the files.
The main inspiration for this git-proj tool came from my desire to use git to version video projects. But video files are HUGE, yet they don't change much. Since most video files are rendered from files that do not change, it is only important to version the video editor's file, so that you can recreate a particular video file version. Since rendering takes time you will want to save the rendered files. But there is usually no need to save every rendered version.
Currently this tool only supports git and large files on a local system. The "remotes" can be setup on mounted drives, and they can be used to clone the files to other systems.
Use "git proj CMD -H usage" to get a quick summary of a command's options.
Use "git prog CMD -h" for full help.
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Output this "long" usage help. See "-H long"
pStyle is used to select the type of help and how it is formatted.
Supported styles:
short|usage - Output short usage help as text.
long|text - Output long usage help as text. All sub-commands.
html - Output long usage help as html. All sub-commands.
md - Output long usage help as markdown.
man - Output long usage help as a man page.
int - Output internal documentation as text.
int-html - Output internal documentation as html.
int-md - Output internal documentation as markdown.
Set verbose (gpVerbose=0) to lowest level: 0
Only very important log messages will be output.
This sets the verbose level (gpVerbose=2) to 2, which is the default.
At level 2, warning and notice messages will be output.
Set the verbose level to N (gpVerbose=N)
N
0 - -q,-V0 critical, errors, and important warning are output
1 - -V1 warnings and above are output
2 - -v,-V2 notice and above are output
3 - -V3 info and above are output
The verbose level can also be set with env. var. gpVerbose. However the command line option will override the variable.
gpVerbose and gpDebug control what log messages are output. This is what the log messages look like:
* Normal log messages:
[Command] [warning|notice|info]: [Message] [File:LineNo](ErrCode)
* Error messages (crit will exit, err might continue):
[Command] [crit|err]: Error: [Message] [File:LineNo](ErrCode)
If gpDebug >= 2, a StackTrace will be output.
* An internal error. This is probably a defect in the code (collect all the output for a bug report):
[Command] [crit|err]: Internal: Error: [Message] [File:LineNo](ErrCode)
StackTrace: ...
Key:
[Command] - the top level command
[crit|err|warning|notice|info|debug] - log levels
[File:LineNo](ErrCode) - exactly where the error message came from (optional)
Set the gpDebug level number. Add 1 for each -x argument. Or you can set gpDebug before running the script. Or you can use the -X option.
Set the gpDebug level to N. The command line options will override the gpDebug env. var.
0 - no debug messages
>0 - "debug" messages
1 - "debug-1" messages
2 - "debug-2" and "debug-1" messages
...
N - "debug-N" and messages less than N
Debug log messages look like this::
[Command] [debug]: debug-N: [Message] [File:LineNo](ErrCode)
The sub-commands will return 0, if there were no serious errors. Even if 0, pay attention to the log messages that are output.
If a sub-command retuns a non-0 code, then some change is needed before trying again. Hopefully the error and warning message describe what needs to change.
See the tutorial document for some examples.
HOME, USER, gpSysLog, gpFacility, gpVerbose, gpDebug, gpAuto
See /usr/share/doc/git-proj/user-doc/config.html for a complete list of env. vars.
See /usr/share/doc/git-proj/user-doc/config.html for details about these files and all the important gitproj variables. The pre-commit hook and its config vars are also described.
All the User Documentation can also be found online in github at: /doc/user-doc
These are the main configuration files you will need to know about.
/usr/share/doc/git-proj/config/gitconfig # Product
~/.gitconfig # User (--global)
PROJ/.gitproj # Project
PROJ/.git/config # Local (--local)
/usr/share/doc/git-proj/config/gitignore # Product
~/.gitignore # User
PROJ/.gitignore # Project
/usr/share/doc/git-proj/hooks/pre-commit # Product
~/.pre-commit # User
PROJ/.pre-commit # Project
PROJ/.git/hooks/pre-commit # Local
git proj
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj config
git proj add TODO
git flow
/usr/share/doc/git-proj/user-doc/
git-proj.html - all sub-commands in one file (html format)
git-proj.md - all sub-commands in one file (markdown format)
config.html - important configuration information
config.md - important configuration information
subcommands/
git-proj-CMD.html
git-proj-CMD.md
tutorial/
*.html
*.md
Online docs: L<https://github.com/TurtleEngr/gitproj/tree/main/doc|https://github.com/TurtleEngr/gitproj/tree/main/doc>
Currently gitproj only supports local git repos. The repos are called "remote" but that is only because the repo could be put on a "mounted" disk. Only the "origin" remote is supported. However each workspace can have its own "origin" definition, because the mount points could be different between systems.
An existing remote git repo can be used, BUT manual work will be needed to set it up. A tutorial may be created to describe the manual changes.
A future implementation could support git repos that are truly remote, on other systems. When that is implemented, an existing repo could be "upgraded" to be a gitproj repo. The "raw" file remote could also be saved on other systems (via rsync, rclone, or even cvs).
The list of feature requests can be found at: enhancements
Use use the -x or -X options, or gpDebug env. var. to turn on debug levels. Larger numbers, more debug. There is only a little bit of debug output, because the code uses "test code" to identify problems, before the code is released. See the github repo for the development process.
Please report bugs at: issues.
Use the "Bug Report" or "Feature Request" templates to submit your issue.
TurtleEngr
(c) Copyright 2022 by TurtleEngr
git proj init -l pLocalDir [-a] [-s pMaxSize] [-m] [-f]
[common-options]
This will create a local git repo with branches. If git-flow is installed can be setup too. After "git proj init" is done, run "git proj remote" to setup remote for git and raw files.
If there is a -a option, "git proj init" will be run with all the default options, which can be overridden with other options.
If there is no -a option, you will be prompted for the settings. See the OPTION section for details.
When the local and remote git repos are setup, all the settings will be saved to [project]/.git/config and [project]/.gitproj
The -a option will automatically run the whole init process with default settings. The other options can be defined to override the default settings.
Define the existing project directory. The last directory will be used for the name of the project. Required.
Prompt:
Dir (-l) [$PWD]? $gpLocalTopDir
So the project Name will be: ${gpLocalTopDir##*/}
Define the "size" for large binary files. Default 10K
Prompt:
Size (-s) [10K]?
Prompt:
Binary files greater than [pSize] were found in your project dir:
[file list]
The listed files can be moved to the project's "raw" directory. Dirs will be created in the raw directory that correspond to the project's directory. A symlink will replace the moved file. The symlink will point to [raw].
The symlinks are only provided for backward compatibility; it would be best to remove those links and modify your code and apps to access the files directly from the raw directories.
Prompt:
Move the files (-m) [y/n]?
If git-flow is installed.
Prompt:
Setup git-flow (-f) [y/n]?
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details. Or "man git-proj" for help with all the sub-commands.
0 - if OK
!0 - if errors
git proj init -l pLocalDir [-a] [-s pMaxSize] [-m] [-f]
For this example, you have a directory of files at:
~/project/bigsur-video/
bigsur.kdenlive
src/
MVI_0224.MP4
MVI_0225.MP4
You have started editing, when you realize you should version the bigsur-vacation files. To do this automatically (-a, no prompts), quietly (-q), move binary files greater than 10K (-m), and add git-flow configs (-f).
cd ~/project/bigsur-video
git init -l $PWD -aqmf
This is what the bigsur-video/ will look like after:
~/project/bigsur-video/
.gitproj
.gitignore
.pre-commit
raw/
src/
MVI_0224.MP4
MVI_0225.MP4
.git/
config (configs copied from .gitproj)
hooks/
pre-commit (copied from .pre-commit)
[other-dirs]/
bigsur.kdenlive
src/
MVI_0224.MP4 -> ../raw/MVI_0224.MP4
MVI_0225.MP4 -> ../raw/MVI_0225.MP4
And if this is the first time you have run a git-proj command, these files will be created (or merged with the files) in in your home dir:
$HOME/
.gitconfig (see the gitproj and gitflow sections)
.gitignore
.pre-commit
Or to be prompted, do this:
cd ~/project/bigsur-video
git init -l $PWD
Now you can used the usual git commands to save your changes for the files that are not in raw/. To save all the files to an external drive, see the git proj remote command (and the push/pull commands).
git proj
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj config
git proj add TBD
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
git proj remote [-a] [-d pMountDir] [common-options]
This will create a remote git repo on an external drive.
(Future versions may support repos on remote computers, where you have rsync access.)
This command is run after "git proj init" to setup a remote location. After this is setup, "git proj push" and "git proj pull" can be used to push/pull git and raw file changes.
Export the git repo to an external drive (or another local dir) This is usually the removable drive's "top" directory. Ideally the top directory should be different across a set of external drives, so that the local "origin" can be used to make sure the proper git repo is found on the drive. The git "origin" will be set to pMountDir/ProjName.git And "remote-raw-origin" will be set to pMountDir/ProjName.raw
-h (-H text)
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
-y | -n (only used with -a option)
Run "git proj -h" for details.
0 - if OK
!0 - if errors
This example direcory is what you have after running git init:
~/project/bigsur-video/
.gitproj
.gitignore
.pre-commit
raw/
src/
MVI_0224.MP4
MVI_0225.MP4
.git/
config (configs copied from .gitproj)
hooks/
pre-commit (copied from .pre-commit)
[other-dirs]/
bigsur.kdenlive
src/
MVI_0224.MP4 -> ../raw/MVI_0224.MP4
MVI_0225.MP4 -> ../raw/MVI_0225.MP4
You want to save these files to an external drive so that they are backed up and so that you can clone them to another computer. Or to restore them, if you remove the project from your computer.
For this example, assume the external drive is mounted at /mnt/usb-drive/ and that there is a top directory "video-proj" on it. This is the quick quiet way of defining the remote location for the project files:
cd ~/project/bigsur-video/
git proj remote -aqd /mnt/usb-drive/video-proj
This is what will be created on the mounted drive:
/mnt/usb-drive/video-proj/
bigsur-video.raw/
src/
MVI_0224.MP4
MVI_0225.MP4
bigsur-video.git/
config
hooks/
pre-config
[other-dirs]/
git config --get remote.origin.url
outputs:
/mnt/usb-drive/video-proj/bigsur-video.git
git config --get gitproj.config.remote-raw-origin
outputs:
/mnt/usb-drive/video-proj/bigsur-video.raw
See git proj push/pull for how to push or pull files to/from the external drive.
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj add
git proj config
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
git proj clone -d pRemoteDir [-a] [-y|-n] [common-options]
Clone a local git repo from the external pRemoteDir.
Use this script get a local copy of the remote git repo frome a mounted drive.
TODO
Automated mode. Use this in batch (non-interactive) scripts. See the -y|-n options.
These are only used if -a option is defined. Where there are "exceptions" these will be used for the default answer: -y continue, -n quit.
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details.
0 - if OK
!0 - if errors
The drive is mounted at: /mnt/usb-video and the the repo is at: video-2019-04-01/trip.git So use this to get a local copy.
proj-get-local -d /mnt/usb-video/video-2019-04-01/trip.git
git proj
git proj remote
git proj clone
git proj add
git proj push
git proj pull
git proj set
git proj status
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
git proj push [-g] [-d] [-a] [-y|-n] [common-options]
rsync will be used top copy the '[ProjName]/raw/' files to
[remote-raw-origin]/[ProjName].raw.
If the -g option is given then run:
"git push origin [current-branch]"
If the -d option is used, then the remote raw/ will be made identical to the local raw/. I.e. files might be deleted from the remote raw/.
See the EXAMPLES section for a "safe" way to use this option.
This turns on automated push for raw/ files. Use the -y or -n to select the action. -n will just display what would be done.
If there is a -d option, that will be removed and the -n option will be added. -d could be very destructive, so it needs to be run interactively.
If -y, then push files to the remote raw/
If -n, then just show what would be pushed to the remote raw/
Default: -n
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details.
0 - if OK
!0 - if errors
The rsync (or rclone) updates of the files in raw/ are not versioned, so be very careful with the -d option; files will be permanently deleted.
For a "safe" way, of updating the files in raw/: do a "push", then "pull" with no -d. Then manually remove files you don't want, in the local raw/ dir. Now run "push" with -d, to update the remote raw/
For example:
cd PROJ
mount REMOTE-DRIVE
# Update remote with all local raw/ files
git proj push
# Get all remote raw/ files
git proj pull
# Review and remove fiies from local, preparing for permanent removal
cd raw
rm FILES
# Remove raw/ files from remote, that are not in local raw/ dir
git proj push -d
Note: Because is is so destructive, the -d option cannot be used with the -a option. Also the -y option will be ignored--you must answer the "delete?" prompt.
To quickly and quietly push files to raw and git you would run this
git proj push -gay
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj add
git proj config
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
git proj pull [-g] [-d] [-a] [-y|-n] [common-options]
rsync will be used top copy the [remote-raw-origin]/[ProjName].raw/ files to '[ProjName]/raw/'.
If the -g option is given then run:
"git pull origin [current-branch]"
If the -g option is given then run:
"git push origin [current-branch]"
If the -d option is used, then the local raw/ will be made identical to the remote raw/. I.e. files might be deleted from the local raw/.
See the help EXAMPLES section, in "git proj push", for a "safe" way to use this option.
This turns on automated pull for raw/ files. Use the -y or -n to select the action. -n will just display what would be done.
If there is a -d option, that will be removed and the -n option will be added. -d could be very destructive, so it must be used interactively.
These are only use with the -a option.
If -y, then pull files from the remote raw/
If -n, then just show what would be pulled from the remote raw/
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details.
0 - if OK
!0 - if errors
cd PROJ
git proj pull
(Error: remote not mounted)
mount REMOTE-DRIVE
git proj pull
To quickly and quietly pull raw/ and git files from remote, use this:
git proj pull -gay
Note: remote raw files will be merged with the local raw files, but no local files will be removed. To make raw/ identical to remote raw/, i.e. allow deletes in local dir use the -d option.
git proj pull -d
git proj
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj config
git proj add TBD
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
git proj status [-g "pGitOpt"] [-r "pRawOpt"] [common-options]
Do a "git status"
Verify gitproj.config.remote-raw-origin is defined and mounted.
Verify origin is set to a path that exists (if mounted).
Give a "diff" (-qr) of the raw files, local vs remote (if mounted)
Git status options. For example to give a "short" status and show ignored files, use::
git proj status -g "-s --ignored"
See git status --help for the status options.
Raw options. Currently these options will be passed to the diff command. For example to show the raw files that are the same between local and remote:
git proj status -r "-s"
See git diff --help for the diff options.
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details. Or "man git-proj" for help with all the sub-commands.
0 - if OK
!0 - if errors
git proj
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj config
git proj add TBD
git flow
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr
NOT IMPLEMENTED YET
git proj config [common-options]
This is a text menu driven tool for checking and "tuning up" your git-proj configurations. There are 4 configuration "levels" to consider: Product, User, Project, and Local. The standard git support configs at the User and Local levels (--global and --local) can be used with "git config" to get and set variables. Or you can edit the files directly. See the FILES section for the files listed by their levels.
The "git proj config" tool has two major sections: "Health Checks" and "Actions".
The "problem" health check will report serious problems that should be fixed. The other health checks are useful for finding areas that you might want to update. For example, your user level configs may have changed and you might notice some project level configs are different. You can then run an "action" to make them consistent.
The "actions" are used to copy config variables and files between the different "levels". Some actions will only copy things that are missing, while other actions can "force" changes.
ProductConfig /usr/share/doc/git-proj/config/gitconfig
UserConfig ~/.gitconfig (--global)
ProjectConfig PROJ/.gitproj
LocalConfig PROJ/.git/config (--local)
ProductIgnore /usr/share/doc/git-proj/config/gitignore
UserIgnore ~/.gitignore
ProjectIgnore PROJ/.gitignore
ProductPreCommit /usr/share/doc/git-proj/hooks/pre-commit
UserPreCommit ~/.pre-commit
ProjectPreCommit PROJ/.pre-commit
LocalPreCommit PROJ/.git/hooks/pre-commit
In the lower level menus, #1 will change to "Back" and "Quit" will move to #3.
This will be available in all menus.
These are done even if not in a git-proj managed workspace.
The Validate Checks, check ProductConfig and --global.
Validate "facility" name [error]
Validate all bool vars are only true/false [error]
Validate all int vars only have numbers [error]
Compare --global with ProductConfig [diff]
Compare UserIgnore with ProductIgnore [diff]
Compare UserPreCommit with ProductPreCommit [diff]
if git-flow=true
is git-flow installed? [warning]
are git-flow section vars defined in --global? [warning]
These are only done if you are in a git-proj managed workspace.
The Validate Checks, check ProjectConfig and --local.
If defined, validate "facility" name [error]
Validate all gitproj bool vars are only true/false [error]
Validate all gitproj int vars only have numbers [error]
Compare ProjectConfig with --global (only list the diffs from all gitproj sections and the diffs that are in .gitproj and also in --global [diff]
Compare ProjectConfig with --local (only list the diffs from all gitproj sections and the diffs that are in ProjectConfig and also in --global [diff]
Compare ProjectIgnore with UserIgnore [diff]
Compare ProjectPreCommit with UserPreCommit [diff]
Compare LocalPreCommit with ProjectPreCommit [diff]
Note the disabled pre-commit checks [info]
gitproj.config.proj-name != the git top dir name [warning]
gitproj.config.local-status - is TBD [error]
remote.origin.url - is not defined [warning]
"git proj init" needs to be run.
gitproj.config.remote-status - is TBD [error]
"git proj remote" needs to be run.
gitproj.config.remote-raw-origin - is TBD [warning]
"git proj remote" needs to be run.
warn if gitproj.config.remote.min-space is < 2x existing space [warning]
if gitproj.config.remote-raw-origin, report if not found. [warning]
Suggest mounting it or running "git proj remote".
List symlinks with problems [error]
Report on all symlinks in PROJ [info]
Check for big binary files, not in raw/ [warning]
This will list all of the current config var settings and what file they are set in. Other statistics about the course will also be listed, for example, the number of files in raw/ and the space they use.
Define the config sections or files that you want to update or copy."
These options are for all of the config files actions.
Define what level to copy from and what level to copy to.
For example, if you select 'Product -> User', then that means sections from /usr/share/doc/git-proj/config/gitconfig will be copied to ~/.gitconfig
The config 'levels' and files are more completely described in the user-doc 'gitproj Configuration Documentation' (config.md
If you select 'Force copy', then the variables in the 'from' file will replace the variables in the 'to' file.
If you select 'Only copy missing', then existing variables, in the 'to' file, will not be replaced with variables in the 'from' file. But missing variables will be copied from the 'from' file.
Define what level to copy from and what level to copy to.
For example, if you select 'Product -> User', then that means /usr/share/doc/git-proj/config/gitignore will be merged to ~/.gitignore
An existing ~/.gitignore will be copied to ~/.gitignore.bak The updated ~/.gitignore will be sorted with duplicates and comments removed.
The config 'levels' and files are more completely described in the user-doc 'gitproj Configuration Documentation' (config.md)
Define what level to copy from and what level to copy to.
For example, if you select 'Product -> User', then that means /usr/share/doc/git-proj/hooks/pre-commit will be copied to ~/.pre-commit
An existing ~/.pre-commit will be copied to ~/.pre-commit.~1~
The config 'levels' and files are more completely described in the user-doc 'gitproj Configuration Documentation' (config.md)
Selecting 'Yes' will make the changes you have selected. If you do not want to continue, then you can Quit, to return to the main menu, or select the Back options to update your selections.
Are the above actions correct?
-h
-H usage|text|html|md|man|int|int-html|int-md
-q, -v, -V N (gpVerbose)
-x, -xx..., -X N (gpDebug)
Run "git proj -h" for details. Or "man git-proj" for help with all the sub-commands.
0 - if OK
!0 - if errors
ProductConfig is short for /usr/share/doc/git-proj/config/gitconfig
ProductIgnore is short for /usr/share/doc/git-proj/config/gitignore
ProductPreCommit is short for /usr/share/doc/git-proj/hooks/pre-commit
--global is short for ~/.gitconfig
UserIgnore is short for ~/.gitignore
UserPreCommit is short for ~/.pre-commit
(Note: this may change to: ~/.git/hooks/pre-commit)
ProjectConfig is short for ProjName/.gitproj
ProjectIgnore is short for ProjName/.gitignore
ProjectPreCommit is short for ProjName/.pre-commit
--local is short for ProjName/.git/config
LocalPreCommit is short for ProjName/.git/hooks/pre-commit
git proj
git proj init
git proj remote
git proj clone
git proj push
git proj pull
git proj status
git proj config
git proj add TBD
git flow
The checks and actions do not cover "all" of the configuration changes that you might like. "git proj config" is just a helper tool to point out problems and differences that might matter.
You are free to edit the files directly. Just make sure the bool and int vars are syntactically OK. Also the Product level configs should probably be left unchanged, so that git-proj updates will not overwrite your changes. Put the changes you want to keep at the --global or Project Levels.
TurtleEngr
GPLv3 Copyright 2021 by TurtleEngr