Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Serverless Application
This is a serverless application using AWS SAM.
Prerequisite
- pyenv
- aws-cli
- docker
- direnv
Installation
git clone https://github.com/AlisProject/serverless-application.git
cd serverless-application
pyenv install
# libraries
python -m venv venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_test.txtEnvironment valuables
# Create .envrc to suit your environment.
cp -pr .envrc.sample .envrc
vi .envrc # edit
# allow
direnv allowTest
Set up dynamoDB local
Download and unzip the dynamoDB local zip in any directory
For example
$ curl -O https://s3-ap-northeast-1.amazonaws.com/dynamodb-local-tokyo/dynamodb_local_latest.tar.gz
$ tar xf ./dynamodb_local_latest.tar.gz
$ rm ./dynamodb_local_latest.tar.gz
Execute Test
# Start dynamoDB local
java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
# lunch docker for localstackļ¼for MAC OSļ¼
TMPDIR=/private$TMPDIR docker-compose up -d
# lunch docker for elasticsearch
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.2.0
# exec
python exec_test.py
# single directory
python exec_test.py --target /handlers
# ignore specific directory
python exec_test.py --ignore /handlersSet SSM valuables
You have to specify SSM valuables as can as possible.
Deployment via AWS Cloud Formation
Create S3 bucket
aws s3api create-bucket --bucket ${ALIS_APP_ID}-serverless-deploy-bucket \
--create-bucket-configuration LocationConstraint=$AWS_DEFAULT_REGIONPackaging and deployment
Packaging
Pack all resources.
./packaging.shYou can Pack specify resource if you want.
python make_deploy_zip.py --target 'src/handlers/labo/n/random/handler.py'DynamoDB
./deploy.sh database
# Show all tables.
aws dynamodb list-tables |grep ${ALIS_APP_ID}database |sort |tr -d ' ",'And add all of generated table names to SSM.
- See: https://github.com/AlisProject/environment
- You can use
dynamodb-table-replacer.sh
- You can use
Master Data
Add master data to DynamoDB.
./add_master_data.shCognito
./deploy.sh cognitoSpecify generated Cognito User Pool ARN to SSM.
Lambda & API Gateway
You have to add SNS authentication params to SSM.
./deploy.sh function && ./deploy.sh function02 && ./deploy.sh apiElasticSearch
./deploy.sh elasticsearch
# show ElasticSearch Endpoint
aws es describe-elasticsearch-domain --domain-name ${ALIS_APP_ID}elasticsearch | jq '.DomainStatus.Endpoint'
# Notice: After this, Elasticsearch is expensive if it is the default setting, so it may be better to reconfigure its performance settings.And add ElasticSearch Endpoint to SSM.
Add Your local IP to ES access policy.
python elasticsearch-setup.py $(curl https://checkip.amazonaws.com/)Permissions
You have to add RestApiArn, ApiLambdaRole and ElasticSearchEndpoint to SSM.
- See: https://github.com/AlisProject/environment
- You can use
api-stack-replacer.sh
- You can use
./deploy.sh permission You have to update Cognito pre authentication trigger.
# Get function name.
aws lambda list-functions | jq -r --arg FUNCTION \
"${ALIS_APP_ID}api-CognitoTriggerPreAuthentication" '.Functions[] | select(.FunctionName | test($FUNCTION)) | .FunctionName'
# Add it to Cognito via Management consoleFYI:
Lambda & API Gateway are bunch of CloudFormation stacks. You can use the script from next time.
./deploy_api.shFix API settings via a script
# Show generated Rest API ID
aws apigateway get-rest-apis | jq '.items[] | if .name == "'${ALIS_APP_ID}'api" then .id else empty end'
# Set SERVERLESS_REST_API_ID to .envrc
direnv edit
# Show generated Rest API with OAuth ID
aws apigateway get-rest-apis | jq '.items[] | if .name == "'${ALIS_APP_ID}'api-with-oauth" then .id else empty end'
# Set SERVERLESS_REST_API_WITH_OAUTH_ID to .envrc
direnv edit
## Load envs
direnv allow
./fix_api.shSingle API Lambda Function
You can deploy single function on api-template.yaml with using deploy_api_function.py script.
Following example is that ArticlesRecent function is deployed.
python make_deploy_zip.py && ./deploy_api_function.py ArticlesRecentALIS Laboratory resources
Experimental features.
npm ci
npx deployCloudWatch Alarm
For production and staging, you should enable alarms.
./deploy.sh apialarmsCloudfront
For development only. You can create Cloudfront and Route53 resources via Cloudformation.
# Show API Gateway IDs
aws apigateway get-rest-apis | jq -r '.items[] | select( .name | contains("'${ALIS_APP_ID}'")) | .name + " : " + .id'
# Show ACM
aws acm list-certificates --region us-east-1
## Add it to env
direnv edit
# Deployment
./deploy_cloudfront.shResource Groups
You can create Resource Groups if you want.
./create_resource_groups.sh