Tuesday, 28 September 2021

AWS

AWS CloudFormation intrinsic functions

The abbreviation of the intrinsic functions

Fn::Sub is identical to !Sub

Fn::!GetAtt returns the value of an attribute from a resource


Value: !GetAtt 'VM.PublicDnsName' # get VM's Public dns name attribute

Ref returns the value of the specified parameter or resource


VpcId: !Ref VPC

Fn::Sub like string concatenation


TableName: !Sub "${EnvironmentType}-user" # EnvironmentType is a parameter created elsewhere in the CloudFormation template.

Limitation

Lambda

Per region

  • Memory 128M to 10 GB (1MB increments)
  • Max execution time 15 minutes
  • Max size of environment variables 4 KB
  • /tmp 512 MB
  • Max currency 1000
  • Max zip file 50 MB
  • Uncompression file 250 MB

DynamoDB

  • Maximum size of an item is 400 KB

API Gateway

  • Default cache TTL is 300 seconds. Max is 3600 seconds

Saturday, 25 September 2021

Install rails 6.0 in Ubuntu 18

Step 1: remove ruby 2.5 because rails 6 needs ruby 2.6 and up


aptitude purge ruby

Step 2: install ruby

sudo snap install ruby --classic

Step 3: create a symbolic link in /usr/bin

sudo ln -s /snap/bin/ruby /usr/bin/ruby 

Step 4: install rails

sudo gem install rails -v 6.0.2.1 

Step 5: create a symbolic link for rails

sudo ln -s /home/leo2019/.gem/bin/rails /usr/bin/rails 

Step 6: verify

rails -v 

output: Rails 6.0.2.1

Monday, 20 September 2021

npm install and npx

npm install

If it is a new project and does not have package.json, do

npm init

module will be added as dependencies in package.json

npm install express --save 

module will be added as devDependencies in package.json

npm install mocha --save-dev

module will not be added to dependencies list and install it temporarily

npm install express --no-save

If we want to install modules listed in package.json and respect package-lock.json

npm ci 

npx

npx is package runner tool that comes with npm 5.2

  • If module is installed, npx can run command in the bin directory of the module
  • If module is not installed, when you run npx , the module will be installed and then be cleaned

Some examples


npx happy-birthday -u leo
npx create-react-app my-app

For the above examples, we are running happy-birthday and create-react-app runner tools. npx has other functionalities. For details, see Kat Marchán blog for npx

npm upgrade package

install tools to check package

npm install -g npm-check-updates

To check

npx ncu

To fix patches

npx ncu -u -t patch

To fix minor version

npx ncu -u -t minor

To fix main version

npx ncu -u -f react-datepicker 

All these fix command will only change package.json. To instll, run:

 npm install

Small syntax difference ends up different result

Java ArrayList remove function

Assume we have a Integer array list arr


    arr.add(20);
    arr.add(30);
    arr.add(1);
    arr.add(10);
    
    arr.remove(new Integer(1));

    //will return [20, 30, 10]
    System.out.println(arr);
	

    arr.add(20);
    arr.add(30);
    arr.add(1);
    arr.add(10);
    
    arr.remove(1);

    //will return [20, 1, 10]
    System.out.println(arr);
	

In the first case, we are calling remove function from public interface java.util.Collection

public boolean remove(E element)

In the second case, we are calling remove function from public interface java.util.List

public Object remove(int index)

Ruby sort


arr = [3,4,1,0]
arr.sort!
# return [0,1,3,4]
puts arr

arr = [3,4,1,0]
newArr = arr.sort
# return [3,4,1,0]
puts arr

The first case is in house sort. (use sort!). The second case will not change the original list, but return a sorted new list. The only difference is !

Wednesday, 15 September 2021

Amazon DynamoDB

Amazon DynamoDB Indexes

Primary Key

There are two kinds of primary key:

1. Partition key (aslo called hash attribute) only

2. Partition key and sort key. This kind of primary key is also called composite primary key.

The name of partition key comes from the fact that items have the key is mapping to a physical address in storage. For example, in composite primary key case, items which have the same partition key and different sort key are stored together with sorted order by sort key.

Secondary Indexes

1. Global secondary index. Can be created on items which have different partition key. Basically, can use this index to search whole table.

2. Local secondary index. Items need to have the same partition key and different sort key. Most time, this index is useless.

Secondary Index is also a table. You also need to specify primary key(partition key only or composite primary key). Also need to specify projected attributes from the base table. The primary key of the base table will be projected automatically.

Noted: In a DynamoDB table, each key value must be unique. However, the key values in a global secondary index do not need to be unique.

Using Global Secondary Indexes in DynamoDB

Amazon Dynamodb Schema

For attributes used in primary key and indexes, you need to define these attributes. However, you can only define these attributes. In fact, if you define other attributes and try to create a table, you will get error that said attributes do not match. This is because Dynamodb is a schemaless database and you can save attributes you do not define.

This is a good explanation about DynamoDB query how to query DynamoDB

Create Dynamodb table in AWS

One option is to create a cloudformation template for this table. Then deploy template


aws cloudformation deploy --template /path_to_template/my-template.json --stack-name my-new-stack --parameter-overrides Key1=Value1 Key2=Value2 

PHP code snippet to query DynamoDB

//define expression attribute names and expression attribute values. It is a key/value map
//in PHP, we use associate array as a map
//For expression attribute names, need to start with pound sign(#)
//For expression attribute values, need to start with colon(:). Also need to define type.
//Here use marshalValue to add a type. e.g {":agelimit": {"N": 21} }
$attributeNames["#typeAndStatus"] = "typeAndStatus";
$attributeValues[":typeAndStatus"] = self::$marshaller->marshalValue($typeAndStatus);
$attributeValues[":now"] = self::$marshaller->marshalValue(time());

//pass the above expression attribute names and expression attributes values in query
$queryAttributes['ExpressionAttributeNames'] = $attributeNames;
$queryAttributes['ExpressionAttributeValues'] = $attributeValues;

//Therefore, for KeyConditionExpression, we can use expression attribute names and expression attribute values
$queryAttributes['KeyConditionExpression'] = "#typeAndStatus = :typeAndStatus and nextRunTime < :now"

//query the Dynamodb client
$result = self::$client->query($queryAttributes);

DynamoDB

You can set up a DynamoDB local using Dockfile provided by AWS. After set up, you can admin db

Calculation of WCU and RCU

It is speed of size

  • One WCU is 1 KB per second. RCU is 4 KB per second. Also RCU counts on strongly consistent read
  • Item size will be round up
  • For RCU: one strongly consistent read. (== 2 eventaully consistent read)

Global Table

  • Active to Active replication between multiple region
  • Can take up to 2 seconds to do replication
  • aws:rep:updateregion contains the region in which the update was made

See ReplicationLatency Metrics

  • Go to Cloudwatch Service
  • click all metrics tab
  • Choose DynamoDB
  • Choose ReceivingRegion, TableName
  • Find Table and check replicationLatency
  • It will show up in the above chart

Tuesday, 14 September 2021

Python cheat sheet

String


# loop through
for c in "abc":
	print(c)

# string replace with return the end result. It does not change original result
date = datePart.replace("th", "")

# string split to list. X will be [welcome, to, jungle]
txt = "welcome to jungle"
x = txt.split()

# substring. will print 20
datePart = "20th"
print(datePart[0:2])

#substring to the end. will print 0th
datePart = "20th"
print(datePart[1:])

# generate an array for all lower case letters (a...z)
letters = list(string.ascii_lowercase)

#convert a decimal integer to binary string. get "111"
s1 = bin(7).replace("0b", "")

#zero left fill. The length is the length of the result string
s= "123"
# get "00123"
s=s.zfill(5)

#The count() method returns the number of times a specified value appears in the string.
print ("foobar".count("o")) # print 2

# get a char's ascii value. 'a' is 97. For code below, will return 98
print(ord('b'))

#ascii value to char. here print 'a'
print(chr(97))

#find and rfind. return 0 and 4
"abcda".find("a")
"abcda".rfind("a")

# str to int. get 9 for the code below
int("09")

hash


#create hash
hash1 = {}

#initiate hash with default data. use get to set a default value. will print out 0
print (hash1.get('a', 0))

#assigin value
hash1['a'] = 10

# access hash. print out 10
count = hash1['a']
print(count)

# check if key exist in hash. should get false
if 'c' not in hash1:
    print ("not have")
        
# loop through
for k in hash1:
    print(hash1[k])

# clear or empty a hash
hash1.clear()

# get hash keys
hash1.keys()

#decouple with hash
keys = list(pointHash.keys())

array

# append. arr becomes 1,2,4
arr = [1,2]
arr.append(4)

# create two dimension array
		dp = []
        for i in range(n):
            dp.append([10000]*n)

# find index of a element
nums = [2,3,49,1]

try:
	i = nums.index(49)
except ValueError:
	pass
#length
print(len(nums))

# join array into string
" ".join(nums)

# sort. This is in house sort
nums.sort()

# sort two dimention array by x[0], then x[1]
intervals.sort(key=lambda x:[x[0], x[1]])

# sort by x[0]
stockPrices.sort(key=lambda x:x[0])

# reverse in house sort
inventory.sort(reverse=True)

# pop at index
nums.pop(0)

# remove from the begin
nums.pop(0)

# remove from the end
nums.pop()

#initial array with 0
ans = [0] * 100

# initiate array with 0 (another way). The below will get [0,0,0,0,0]
arr=[]
arr = [0 for i in range(5)]

#reverse range
reversed(range(100))

#initialize two dimension array
dp = [[0 for x in range(m)] for y in range(m)] 

# list comprehension
 starts = [a for a, b in flowers]
 ends = [b for a, b in flowers]

Set

# create a empty set
s = set()

# add an element to set
s.add(1)

# check length of set
print(len(s))

# check contains
if 2 in s:

Binary search built in functions

bisect.bisect_left(a, x, lo=0, hi=len(a)) : Returns leftmost insertion
point of x in a sorted list. Last two parameters are optional, 
they are used to search in sublist.

bisect.bisect_right(a, x, lo=0, hi=len(a)) Returns rightmost insertion 
point of x in a sorted list a. Last two parameters are optional,
they are used to search in sublist.

index = bisect_right(arr, val)

Permutation

for p in permutations([1,2,3], 3): 
	print (p)
#output
(1, 2, 3)
(1, 3, 2)
(2, 1, 3)
(2, 3, 1)
(3, 1, 2)
(3, 2, 1)

Rest API

Send a post

import requests
import json

url = 'http://localhost:5000'
myobj = {'somekey': 'somevalue'}
headers = {'content-type':'application/json'}

x = requests.post(url, data =json.dumps( myobj), headers=headers)

print(x.text)

Math

#integer div.It will return 4
9//2

# find gcd of two integers
gcd = math.gcd(d, price)

# val is 8
val = max(3, 8)

#power. mod is optional. But if need mod, use it. It is 
#much faster than ans=pow(x,y) ans = ans%mod
pow(x, y, mod)
ans = ans + pow(2, r-i, 1000000007)

Inner Function


def countDaysTogether(self, arriveAlice: str, leaveAlice: str, arriveBob: str, leaveBob: str) -> int:
    months = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
    def getDayOfYear(dayStr:str)->int:
        arrMonth = int(dayStr[0:2])
        arrDay = int(dayStr[3:5])

Hard questions

1799. Maximize Score After N Operations

class Solution:
    myMap = {}
    def maxScore(self, nums: List[int]) -> int:
        def getM(k, nums):
            # build cache key
            cacheK = ' '.join(str(x) for x in nums)+ "k:" + str(k)
            if cacheK in self.myMap:
                return self.myMap[cacheK]

            if len(nums)==2:
                ans = k*math.gcd(nums[0], nums[1])
                self.myMap[cacheK] = ans
                return ans
            
            ans =0
            l = len(nums)
            for i in range(l-1):
                for j in range(i+1, l):
                    n1 = k*math.gcd(nums[i], nums[j])
                    left = []
                    for h in range(l):
                        if h!=i and h!=j:
                            left.append(nums[h])
                    n2 = getM(k+1, left)

                    if (n1+n2) > ans:
                        ans = n1 + n2
            self.myMap[cacheK] = ans
            return ans

        nums.sort()
        return getM(1, nums)

heap

#Leetcode 2542
class Solution:
    def maxScore(self, nums1: List[int], nums2: List[int], k: int) -> int:
        l = len(nums1)
        nums = []
        for i in range(l):
            nums.append([nums1[i], nums2[i]])
        
        #sort by the second element
        nums.sort(key=lambda x:x[1])
        nums.reverse()

        ans, curSum = 0, 0

        myHeap = []
        for i in range(k):
            curSum += nums[i][0]
            heapq.heappush(myHeap, nums[i][0])
        ans = curSum * nums[k-1][1]

        for i in range(k, l):
            out = heapq.heappop(myHeap)
            curSum = curSum-out + nums[i][0]
            heapq.heappush(myHeap, nums[i][0])
            ans = max(ans, curSum * nums[i][1])

        return ans
#Leetcode 1464
import heapq
class Solution:
    def maxProduct(self, nums: List[int]) -> int:
        for i in range(len(nums)):
            nums[i] = nums[i]*-1
        
        heapq.heapify(nums)
        # return smallest
        a = heapq.heappop(nums)
        b = heapq.heappop(nums)
       
        return (a*-1 -1) * (b*-1-1)

Run codes

#Leetcode 1992
from typing import List
class Solution:
    def findFarmland(self, land: List[List[int]]) -> List[List[int]]:
        def getFarm(f: List[int], a:int, b:int):
            
            if a<f[0] or b<f[1]:
                f[0]=a
                f[1]=b
            if a>f[2] or b>f[3]:
                f[2] = a
                f[3] = b
            return f
            
        rows = len(land)
        cols = len(land[0])

        ones = []

        # find all ones
        for i in range(rows):
            for j in range(cols):
                if land[i][j] == 1:
                    ones.append([i, j])
        
        #put ans
        ans = []
        for one in ones:
            #some one has been grouped
            if land[one[0]][one[1]] != 1:
                continue
            
            # one itself is a farmland
            x, y = one[0], one[1]
            queue = [[x,y]]
            myLand = [x, y, x, y]

            while len(queue)>0:
                size = len(queue)
                for i in range(size):
                    point = queue.pop(0)
                    a, b = point[0], point[1]

                    #left
                    if b!=0 and land[a][b-1] == 1:
                        myLand = getFarm(myLand, a, b-1)
                        land[a][b-1] = 0
                        queue.append([a, b-1])

                    #right
                    if b!=cols-1 and land[a][b+1] == 1:
                        myLand = getFarm(myLand, a, b+1)
                        land[a][b+1] = 0
                        queue.append([a, b+1])

                    #up
                    if a!=0 and land[a-1][b]==1:
                        myLand = getFarm(myLand, a-1, b)
                        land[a-1][b] = 0
                        queue.append([a-1, b])

                    #down
                    if a!=rows-1 and land[a+1][b]==1:
                        myLand = getFarm(myLand, a+1, b)
                        land[a+1][b] = 0
                        queue.append([a+1, b])
            ans.append(myLand)
        
        return ans
        
obj = Solution()
land = [[1,0,0],[0,1,1],[0,1,1]]
res = obj.findFarmland(land)
print(res)

django

commands

django-admin startproject project-name

python manage.py startapp app-name

python manage.py runserver

# open a shell to run some db query
python manage.py shell

Handle Form

# get post data
post_id = request.POST.get("post_id")

Use env variables

# create .env file in the project directory
api_key=mysecfretetemmsmsm

# in settings.py
from dotenv import load_dotenv
load_dotenv()

# to access env variable
import os
os.getenv("api_key")

var dump

from pprint import pprint
pprint(request.POST)

Versions

#find version
python -V
pip -V
python3.7 -m pip list

#show version used to store library
pip show openai

Install Python OpenAi Module

python3.7 -m pip install --upgrade pip
python3.7 -m pip install openai

Sunday, 12 September 2021

Modules in Node.js

Node.js uses CommonJS module format. It uses require to include a module. It uses module.exports to expose a modules. The npm ecosystem also uses this format.

Use require to include a module. The below we include express module.


const express = require('express')
const app = express()

app.get('/', (req, res) => {
  res.send('Hello World!')
})

const port = 2000
app.listen(port, () => {
  console.log(`listen to port 2000...`)
})

Use exports to let other codes to use this module. For example, AWS Lambda function can be written in Node.js

exports.myFunction = (event, context, callback) => {
        //do something
}

The below is a hello world module and how to use it

//in hello-world.js create a module
module.exports = name => console.log("hello to " + name)

//in main.js
var hw = require('./hello-world')
hw("kei")

//run it
node main.js //output: hello to kei
//another example
//in config.js
module.exports = {
  dbPort : 8080,
  dbHost : "etc",
  dbName : "nom",
}

//in main.js
var config = require("./config")
console.log("config:", config);
//more example
//in config
module.exports = {
  dbPort : 8080,
  dbHost : "etc",
  dbName : "nom",
  hello: (name)=> "hello to " + name
}

//in main
var leo = require("./config")

console.log(leo)

console.log(leo.hello("smith"))

//output node main.js
{
  dbPort: 8080,
  dbHost: 'etc',
  dbName: 'nom',
  hello: [Function: hello]
}
hello to smith

However, for ES6, it uses another format. It uses export and import to export and include a module. Here is post about default values in ES6 module

Monday, 6 September 2021

Java Queue

Both PriorityQueue and LinkedList implement Java Queue interface.

There are these methods. Every is a pair. The first one will throw error on failure. The second will return special value instead of throwing error.

  • add(e) offer(e)
  • remove() poll()
  • element() peek()

For LinkedList, we can call removeFirst()

Java PriorityQueue

java PriorityQueue implements java.util.AbstractQueue class which extends java.util.Collection

The head of the queue is the least element with respect to the specified ordering. However, can provide Collections.reverseOrder() in the constructor to create a max heap.

useful methods:

  • peek()
  • poll()

methods from java.util.Collection

  • size()
  • add()
  • isEmpty()

Here is a Leetcode problem leetcode 1046 which can be solved by max heap


import java.util.*;
class Solution {
    public int lastStoneWeight(int[] stones) {
        //create a max heap. The head has the max value
        var q = new PriorityQueue<Integer>(Collections.reverseOrder());

        //add element into heap
        for (int s: stones) {
            q.add(s);
        }

        while(q.size()>1) {
            int x = q.poll();
            int y = q.poll();

            if (x != y) {
            	//put result stone back into queue
                q.add(Math.abs(x-y));
            }
		}

        //check q.size
        if (q.size()==1) {
            return q.poll();
        } else {
            return 0;
        }
    }
}

Tree Level Traversal

The below is a solution for LeetCode 104

public int maxDepth(TreeNode root) {
        if (root ==null) return 0;
        
        var myq = new LinkedList<TreeNode>();
        myq.add(root);
        var ans = 0;
        
        while(!myq.isEmpty()) {
            //queue only holde nodes of current level
            var size = myq.size();
            
            //every iteration is one level because of size
            for(int i=0; i<size; i++) {
                TreeNode cur = myq.poll();
                if (cur.left!=null) myq.add(cur.left);
                if (cur.right!=null) myq.add(cur.right);
             }
            
            ans++;
        }
        
        return ans;
        
    }

Use Paire as queue element

2462

Priority Queue using custom comparator

Leetcode 2099

class Solution {

  public int[] maxSubsequence(int[] nums, int k) {
    var pq = new PriorityQueue<int[]>((a, b) -> a[1] - b[1]);

    for (int i = 0; i < nums.length; i++) {
      int[] t = { i, nums[i] };
      if (pq.size() < k) {
        pq.add(t);
      } else {
        var e = pq.peek();
        if (e[1] < nums[i]) {
          pq.poll();
          pq.add(t);
        }
      }
    }

    int[] index = new int[k];
    int j = 0;
    while (pq.size() > 0) {
      var e = pq.poll();

      index[j] = e[0];
      j++;
    }

    Arrays.sort(index);

    int[] ans = new int[k];
    j = 0;
    for (int i : index) {
      ans[j] = nums[i];
      j++;
    }

    return ans;
  }
}

array as queue element

//See Leetcode 373
PriorityQueue<int[]> minHeap = new PriorityQueue<> ((a, b) -> (a[0] + a[1]) - (b[0] + b[1]));
       
//init. the last two elements are index for two array
minHeap.add(new int[]{nums1[0], nums2[0], 0 ,0});

How to inject codes into blogger

Basically you need to add highlight.js into your blogger template. Then wrapper your code. See highlight.js usage

Here is a good tutorial about that: https://www.youtube.com/watch?v=APNWCe56cQA

Create REST API by AWS API Gateway and AWS Lambda

 We can create a REST API using AWS API Gateway and AWS Lambda by Proxy integration.

For Proxy integration, when API Gateway receives the request, it sends proxy resources such as path information and request body to Lambda function. Lambda function generates a responses based on information sent by API gateway. This response is sent back to API Gateway which send this response back to API user. In Lambda function, you can call other aws services such as store data in s3 bucket and send message to sqs before send back the response.

Step 1: Create a AWS Lamda function

Use Python as example. After create function, need to edit lambda_hander

 
import json

def lambda_handler(event, context):
    # event has info of request passed by API Gateway
    userId = event['queryStringParameters']['userId']

    # build response body
    body = {"user_id": userId, "message": "processed by Lambda"}

    # build response
    response = {'statusCode': 200, 'headers': {}}

    # let API caller know that our body is in json format
    response['headers']['content-type'] = 'application/json'
    response['body'] = json.dumps(body)

    return response

Note for node.js. In node.js, you can do

 
exports.myhandler = {event, context, callback) => {
        //get path parameters
        event.pathParamters...

        //get body
        event.body

        //get query string
        event.queryStringParameters ...

}

Note for debug:

For Python, use print something

For node.js, console.log something

Then we can see these debug info in cloudwatch


Step 2: Create a API Gateway

1. Pick Rest type

2. Create resource. Give it a name

3. Create method. We choose GET for this example.

4. In Get set up. Choose Integration type: Lambda Function. Also check "Use Lambda Proxy integration". Link the lambda function just created.


Step 3: deplay this API Gateway

Step 4: try it

find the api url and append a user id such as ?user_id=1

Step 5: add custom domain

You may want to add custom domain for this api.

Here is a good youtube video about setting up custom domain for API gateway

Here is a good youtube video about Gateway API and lamda function.