
I recently had to query an API for 2 million users and update some data on each of those users. Seems simple enough, but let's break down what's actually going on here. The problem was that I could only query from `/users` in increments of 250, and for each user, another API call to `/users/:user_id/update` needed to be made to update the user information.

### Naive Approach

The easy way to solve this is to query for the 250 users, loop through those users, and send a request to update each user before going to get the next set of 250 users. Only issue here is that this is very time consuming. Even with fast internet and a M1 Mac, I was capping out at around 120 requests per minute, and to get through 2 million users, it would take 277 hours or 11.5 days of constant running. Yikes.

## Crank it with Concurrency

Our computers are surely capable of handling more than one task at a time, so we can leverage that by running the requests concurrently in separate node processes. To help us with this, we're going to employ the use of [BullMQ](https://docs.bullmq.io/), a queueing system built on top of [Redis](https://redis.io/) that will allow us to focus on cranking through our API updates instead of spending hours building out a robust queueing system.

Let's see how that's done...

### Redis Setup

Using BullMQ means we need to have a local (or remote) Redis server running. We just need the server to be running on the same port as BullMQ's queue and worker, so [install Redis](https://redis.io/docs/getting-started/) and toss this into your command prompt:

```shell
$ redis-server --port 5050
```

---

### BullMQ Config

We carry over that port number and tell BullMQ that we want to use a local Redis server with a concurrency factor of 4 (or whatever the API that you'll be blasting can handle). The API I was using had a rate limiter, so the highest I could go to is 4, but I wouldn't use more than the number of cores that your CPU has. A higher number just means more jobs are allowed to be processed in parallel.

```TypeScript
const BULL_CONFIG = {
  connection: { port: 5050, host: 'localhost' },
  concurrency: 4,
}
```

---

### Create the Queue

To initialize our queue, we need an identifier that will be used as a key prefix in the Redis store and the config to connect to Redis. Just like that, the queue is now hooked up and ready to store jobs.

```TypeScript
const queue = new Queue('users-queue', BULL_CONFIG)
```

---

### Start Filling that Queue

We want to fetch the users immediately after starting this script, so we use a recursive IIFY to go through all the paginated pages of users and fetch all 2 million users. We simply loop through all 250 users returned from each request and add a job to the queue using `queue.add()`. We give the job a name based on the type of job that it is, pass it the data that we want the Worker to receive, and use whatever job [options](https://api.docs.bullmq.io/interfaces/v4.BaseJobOptions.html) fit your needs.

```TypeScript
;(async function fetchUsers(pageNum: number = 0) {
  const response = await fetch(`/users?page=${pageNum}`)
  const data = await response.json()
  const users = data.users as User[]

  for (const user of users) {
    // Add each user to the queue as a Job
    await queue.add('update-user', user, { removeOnComplete: true })
  }

  // Check if there are more pages
  if (data.nextPage) await fetchUsers(data.nextPage)
})()
```

---

### Start the Worker

As the jobs continue to be added to the queue, it's time to get working on those jobs. We instantiate a worker with the same identifier of the queue that we used before (`'users-queue'`), giving it a callback function to handle the job when it gets picked up, and passing the same config that we gave the queue so that it's pointed at the correct Redis store.

The callback function we pass the worker accepts a job, which holds the data that we passed it from `queue.add()` (our user), and makes the appropriate fetch request to update that user. With that in place, our worker will handle the job processing (4 at a time given our concurrency setting) and categorizing (with statuses of 'waiting', 'active', 'completed', and 'failed') based on how our fetch request performs.

```TypeScript
const worker = new Worker<User, void>(
  'users-queue',
  async (job: { data: User }) => await updateUser(job.data),
  BULL_CONFIG
)

async function updateUser(user: User) {
  const response = await fetch(`/users/${user.id}/update`)
  const data = await response.json()
  return data.success
}
```

---

### Listen to Worker Completion

We don't need it for our example, but often times there is a need to do something with the job data after the job has been completed, so the worker allows us to setup an event listener for the completed event like so:

```TypeScript
worker.on('completed', async (job: { data: User }) => {
  console.log(`Job ${job.data.id} Completed!`)
})
```

---

## Put it all together

Fully assembled, our script looks like this:

```TypeScript
import { Queue, Worker } from 'bullmq'

const BULL_CONFIG = {
  connection: { port: 5050, host: 'localhost' },
  concurrency: 4,
}

const QUEUE_NAME = 'users-queue'
const JOB_NAME = 'update-user'

type User = {
  id: string
  info: any
}

const queue = new Queue(QUEUE_NAME, BULL_CONFIG)

const worker = new Worker<User, void>(
  QUEUE_NAME,
  async (job: { data: User }) => await updateUser(job.data),
  BULL_CONFIG
)

// Worker callback to update users
async function updateUser(user: User) {
  const response = await fetch(`/users/${user.id}/update`)
  const data = await response.json()
  return data.success
}

// Fetch users and add them to the queue
;(async function fetchUsers(pageNum: number = 0) {
  const response = await fetch(`/users?page=${pageNum}`)
  const data = await response.json()
  const users = data.users as User[]

  for (const user of users) {
    await queue.add(JOB_NAME, user, { removeOnComplete: true })
  }

  if (data.nextPage) await fetchUsers(data.nextPage)
})()

// Listen for job completion
worker.on('completed', async (job: { data: User }) => {
  console.log(`Job ${job.data.id} Completed!`)
})
```

---

## Results

With this setup, the time to update all 2 million users was brought from 11 days down to several hours. Saving energy and time. I hope you learned a thing or two and if you found this article interesting, feel free to hit me up on [GitHub](https://github.com/ChristianAnagnostou) or [X](https://twitter.com/javascramble). Cheers!


Scaling async tasks with concurrency is scary fast

Queueing with BullMQ and Redis


![Window Drawing](/img/articles/zooming-scaling/window-drawing.webp)

Zooming and panning: to two common user interactions that allow users to manipulate and navigate content on a webpage.

Zooming involves changing the scale or magnification level of the content, making it appear larger or smaller. Zooming can be implemented using CSS properties like `transform: scale()` and some added JavaScript to modify the size and layout of the elements dynamically.

Panning involves moving the visible area of content within a fixed viewport. This allows users to explore content that exceeds the available space by dragging or swiping in any direction. Typically, panning is implemented by utilizing `transform: translate(x,y)` with JavaScript to capture user input and adjust the content's position within the container.

---

The image above shows a zoomed out and panned canvas. When initially opening the canvas, the whole screen would be covered with the blue diagonal lines. After zooming out and panning towards the top left, we end up with the layout in the image.

The image also shows the cursor position and the x and y values used to get its position. Those variables can be used to zoom into the location of the cursor. We'll implement that later.

Using the CSS property `transform` with its `scale` and `translate` values, zooming and panning can be combined into one CSS line with two dynamic variables.

```JavaScript
let scale = 1

const zoomPos = { x: 0, y: 0 }

const transform = `scale(${scale}) translate(${zoomPos.x}px, ${zoomPos.y}px)`
```

---

## Panning

To pan, all we need to do is change the `zoomPos` values based on how far the mouse moves as well as the direction in which it moved. Do this by listening for the `mousedown`, `mousemove` and `mouseup` events. Mouse down and mouse up only toggle a variable to let the mouse move event know that the mouse is pressed while moving - dragging, so the bulk of the work is done by mouse move.

To move the initial window, start by attempting to use a stored value, `prevMouseMove`, which at first doesn't exist, so default to the current mouse position. Subtract the first non-null value of those two from the current mouse position and the result is the distance that the mouse moved. Finally, add that to the current zoom position.

```javascript
let prevMouseMove = null

if (mouseDown) {
  zoomPos.x += clientX - (prevMouseMove?.clientX || clientX)
  zoomPos.y += clientY - (prevMouseMove?.clientY || clientY)
}

prevMouseMove = { clientX, clientY }
```

---

## Zooming

The intended user experience for zooming depends on the application, but I find it nice when you zoom in and the window hones in on the location of the cursor, so we'll implement that approach. To do that, both scale and transform need to be updated since the window must move in relation to the cursor while zooming in/out.

### Calculating Scale

The steps to calculate the scale are:

1. Set some config variables
2. Determine the zoom direction (in or out)
3. Adjust the scale based on zoom direction

#### Config Vars

The config variables allow us to control the behaivor of the zooming:

```javascript
const maxScale = 4 // Max zoom (400%)
const minScale = 0.1 // Min zoom (10%)
const zoomFactor = 0.05 // Zoom speed (higher is faster)
```

#### Zoom Direction

Typically, zooming is handled by the scroll of the mouse wheel, so we listen for the `wheel` event on the `window`. The window wheel event contains information about the direction of the wheel (or trackpad) and, depending on the browser, the information varies in output. The safest way I've found to capture wheel direction is by clamping the scroll between -1 and 1 so that the positive one is zooming in and the negative one is zooming out.

```javascript
const delta = e.wheelDelta || e.originalEvent.detail // Chrome || Firefox

const clampedDelta = Math.max(-1, Math.min(1, d))
```

#### Scale

To calculate the new scale, we use our current scale multiplied by the zoom factor to get the next _step_ of zoom. We then multiply that by the zoom direction (-/+). Finally, we add that to the current scale and clamp the result between the min and max scale config variables.

```JavaScript
const scale = scale + (delta * zoomFactor * scale)

const clampedScale = Math.max(minScale, Math.min(maxScale, scale))
```

---

### Calculating Transform

Now that the scale has been adjusted, we can use it to modify the values of zoom position x and y. We need to first get the distance between the cursor and the center of the window. Then that same distance needs to be removed from the previous zoom position and scaled down to a px value relative to the initial window.

If that was a lot, picture it like this...

Imagine being zoomed out really far with the initial window as a small square in the center of the screen. If you were to click at the top left of the screen it may be just a few hundred pixels away from the initial window, but if you were to zoom back in and try to click the same spot on the canvas, it would be thousands of pixels up and to the left. That's what screenX and screenY represent.

The next step would be to update the scale value described in the section above. With scale being updated, move the window towards the location of the cursor keeping the relative distance from the center of the screen constant.

```javascript
const zoomPointX = e.clientX - window.innerWidth / 2
const zoomPointY = e.clientY - window.innerHeight / 2

const screenX = (zoomPointX - zoomPos.x) / prevScale
const screenY = (zoomPointY - zoomPos.y) / prevScale

... // update scale value

zoomPos.x = -screenX * scale + zoomPointX
zoomPos.y = -screenY * scale + zoomPointY
```

Note: `prevScale` doesn't refer to a separate variable, it's just showing that they will be different values.

---

## All Together

Combining the zoom scale and transform into one event callback looks like the code below. There are several similar lines that could be extracted to auxiliary functions, but I'll leave that to you for the sake of keeping actions contained.

```typescript
let scale = 1
const zoomPos = { x: 0, y: 0 }

const maxScale = 4 // Max zoom (400%)
const minScale = 0.1 // Min zoom (10%)
const zoomFactor = 0.05 // Zoom speed (higher is faster)

let prevMouseMove = null

canvas.addEventListener('mousedown', ({ clientX, clientY }: MouseEvent) => {
  prevMouseMove = { clientX, clientY }
})

canvas.addEventListener('mouseup', () => {
  prevMouseMove = null
})

canvas.addEventListener('mousemove', ({ clientX, clientY }: MouseEvent) => {
  if (prevMouseMove) {
    // Pan canvas by distance from prevMouseMove
    zoomPos.x += clientX - (prevMouseMove?.clientX || clientX)
    zoomPos.y += clientY - (prevMouseMove?.clientY || clientY)
  }

  // Update to track next distance change
  prevMouseMove = { clientX, clientY }
})

window.addEventListener('wheel', (e: Event) => {
  const zoomPointX = e.clientX - window.innerWidth / 2
  const zoomPointY = e.clientY - window.innerHeight / 2

  const screenX = (zoomPointX - zoomPos.x) / scale
  const screenY = (zoomPointY - zoomPos.y) / scale

  // Determine if zooming in or out
  const d = e.wheelDelta || e.originalEvent.detail // Chrome || Firefox
  const delta = Math.max(-1, Math.min(1, d)) // Cap the delta to [-1,1] for cross browser consistency

  const updatedScale = scale + delta * zoomFactor * scale
  scale = Math.max(minScale, Math.min(maxScale, updatedScale))

  zoomPos.x = -screenX * delta + zoomPointX
  zoomPos.y = -screenY * delta + zoomPointY
})
```

## Wrap Up Notes

The important thing to remember about zooming is that there are two types of pixel _units_: canvas and screen. Converting between the two is often needed when building on a surface that scales. Knowing when to use each unit is key to understanding the relationship between the screen and the canvas.

Below I've added my implementations to convert from canvas to screen and vice versa. This comes from an open source project of mine called [Qwikdraw](https://www.qwikdraw.app/) where you can draw shapes and add images to a canvas and orient, rotate, and modify them.

```typescript
const screenToCanvas = $((screenX: number, screenY: number) => {
  return {
    canvasX: (screenX - zoomPos.x - (window.innerWidth / 2) * (1 - scale)) / scale,
    canvasY: (screenY - zoomPos.y - (window.innerHeight / 2) * (1 - scale)) / scale,
  }
})

const canvasToScreen = $((canvasX: number, canvasY: number) => {
  return {
    screenX: (canvasX - zoomPos.x) / scale,
    screenY: (canvasY - zoomPos.y) / scale,
  }
})
```

---

Hope this article was helpful to you! If it was, let me know on Twitter [@javascramble](https://twitter.com/javascramble).


Turn your browser window into a 2D canvas that you can explore

Zooming & Panning


There comes a time in every frontend dev's career when they will need to render large amounts of data in list form. The simple approach is to loop through all items and create the HTML for each. That can be thousands of elements when you have a lot of data, so it's better to show only a subset of those items through a process called virtualization. Using a virtual list allows you to render the data without making your site feel laggy and resource intensive. The goal is to render only the visible portion of the list of items and dynamically swap items in and out of the DOM as the user scrolls. Doing so allows for a more efficient rendering process and better user experience.

---

### How to make one

To operate this implementation of a virtual list, there are a few limitations you should know. Each element in the list needs to be the same height and the height of the virtual list needs to be defined outside of the virtual list component (we'll go over a nifty trick for this at the end). With that in mind, here are the variables that need to be passed into the component:

### `numItems`

Number of elements that are in the list

### `itemHeight`

Height of the element in pixels

### `windowHeight`

Height of the wrapping element in pixels

### `renderItem({ index, style })`

Function that accepts the element index and style and returns a JSX element

### `overscan`

Number of elements above or below the screen that will pre-render (helps when scrolling really fast)

---

### Prop Types

With those parameters, we're able to construct our virtual list.
If we were to write them as TypeScript props for a React virtual list component, it would look something like this:

```tsx
type Props = {
  numItems: number
  itemHeight: number
  windowHeight: number
  renderItem: ({ index, style }: { index: number; style: React.CSSProperties }) => JSX.Element
  overscan?: number
}
```

---

### Internal State

The virtual list itself only needs to keep track of one variable: [scrollTop](https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollTop). This is the number of pixels that an element's content is scrolled vertically. The initial value is 0 and would look like this:

```ts
const [scrollTop, setScrollTop] = useState(0)
```

---

### Calculate Starting and Ending Elements

The next step is to calculate the starting and ending indexes for the elements that should render.
The starting index is found by taking the scroll top position and dividing it by the height of the elements to get the index of the element at the top of the scroll container. From there all you do is subtract the overscan variable and you have the starting index to render. To get the ending index you add the scroll top position to the height of the scroll container and divide that by the height of the elements. Add the overscan this time and boom, there's your end index.

```ts
const startIndex = Math.max(0, Math.floor(scrollTop / itemHeight) - overscan)

const endIndex = Math.min(numItems - 1, Math.floor((scrollTop + windowHeight) / itemHeight) + overscan)
```

---

### Create List of Elements for Render

With our newly created index variables, we now know which range of elements to render. To do that, we can use a simple for loop and call the `renderItem` function being passed into the virtual list component. We make each element positioned absolutely and calculate its top position by its index in the list. This should return a JSX element and we add that to the items array to prep them for rendering.

```tsx
const items: JSX.Element[] = []

for (let i = startIndex; i <= endIndex; i++) {
  items.push(
    renderItem({
      index: i,
      style: { position: 'absolute', top: `${i * itemHeight}px`, width: '100%' },
    })
  )
}
```

---

### Render the elements

The JSX markup for the virtual list is quite minimal. You need an outer scrolling element and an inner element whose height is calculated by the total number of list elements multiplied by the height of the elements. The array of elements being rendered can be added directly into the markup as it's just an array of JSX elements.

```ts
const innerHeight = numItems * itemHeight
```

```tsx
<div className="scroll" style={{ overflowY: 'scroll', width: '100%' }} onScroll={onScroll}>
  <div className="inner" style={{ position: 'relative', height: `${innerHeight}px` }}>
    {items}
  </div>
</div>
```

You might have noticed that there is an `onScroll` property on the outer scrolling element. This is to track the scrollTop of the outer element and update the scrollTop state.

```ts
const onScroll = ({ currentTarget }) => setScrollTop(currentTarget.scrollTop)
```

---

### Fully Assembled

When we put all those steps together, you get a reusable component that will help you render looooong lists of data. There are limitations with this simple setup but you can modify it to fit your needs. Here is the code to do so:

```tsx
import React, { useState } from 'react'

type Props = {
  numItems: number
  itemHeight: number
  windowHeight: number
  renderItem: ({ index, style }: { index: number; style: React.CSSProperties }) => JSX.Element
  overscan?: number
}

const VirtualList = (props: Props) => {
  const { numItems, itemHeight, renderItem, windowHeight, overscan = 3 } = props

  const [scrollTop, setScrollTop] = useState(0)

  const innerHeight = numItems * itemHeight
  const startIndex = Math.max(0, Math.floor(scrollTop / itemHeight) - overscan)
  const endIndex = Math.min(numItems - 1, Math.floor((scrollTop + windowHeight) / itemHeight) + overscan)

  const items: JSX.Element[] = []

  for (let i = startIndex; i <= endIndex; i++) {
    items.push(
      renderItem({
        index: i,
        style: { position: 'absolute', top: `${i * itemHeight}px`, width: '100%' },
      })
    )
  }

  const onScroll = ({ currentTarget }) => setScrollTop(currentTarget.scrollTop)

  return (
    <div className="scroll" style={{ overflowY: 'scroll', width: '100%' }} onScroll={onScroll}>
      <div className="inner" style={{ position: 'relative', height: `${innerHeight}px` }}>
        {items}
      </div>
    </div>
  )
}

export default VirtualList
```

---

### Using the Virtual List

Since the virtual list needs its height defined, you can wrap it in an element with a React ref and use the clientHeight of the ref to make the height responsive to whatever the height is of its wrapping element.

The element being rendered is found by referencing the index in the list of all data and the styles are added inline on the list element itself.

```tsx
<div ref={virtualListContainer}>
  <VirtualList
    numItems={allData.length}
    itemHeight={30}
    windowHeight={virtualListContainer.current?.clientHeight || 0}
    renderItem={({ index, style }) => (
      <div key={allData[index].id} style={style}>
        {allData[index].title}
      </div>
    )}
  />
</div>
```

---

Hope you enjoyed reading through this article. If you happen to need a virtual list for Qwik, [here's my virtual list implementation](https://github.com/christiananagnostou/jukebox/blob/master/src/components/Shared/VirtualList.tsx) - it includes a few other added features that make for a nice utility component.


Ever tried scrolling down on a table with 100,000 rows? Good luck. Virtual Lists unlock that ability.

Virtual Lists are Beautiful


---

In 1868, a man by the name of [Christopher Sholes](https://en.wikipedia.org/wiki/Christopher_Latham_Sholes) was sitting at his desk with this sequence of letters and numbers in front of him:

```
 3 5 7 9 N O P Q R S T U V W X Y Z
2 4 6 8 . A B C D E F G H I J K L M

```

It was the keyboard of the first ever typewriter.
The letters were staggered resembling a piano, had black keys and white keys, and was laid out in two long rows. There were no keys for 0 or 1 as the letters O and I were deemed sufficient.

Eventually, after years of experimenting with different layouts to make the keys jam less, Sholes ended up with this layout:

```
 1 2 3 4 5 6 7 8 9 0 - =
Q W E R T Y U I O P [ ] \
 A S D F G H J K L ; '
  Z X C V B N M , . /

```

Being a newspaper publisher, Sholes would spend hours typing up new documents, so jamming the typewriter usually meant smudges or restarting altogether. Sholes realized that all he had to do was rearrange the keys and spend some time getting used to the layout for the jamming to stop.

Keeping the same mindset, let's apply this to our work today. Many of us rely on a keyboard to do our jobs, most look exactly like Sholes', and the more we practice using it, the faster and more accurate we get. That alone is a useful skill, but why not lean into those impressive keyboard skills a little more for some neat added benefits.

### Shortcuts

Keyboard shortcuts are great because they allow you to perform tasks quickly and efficiently, without having to use a mouse or trackpad to navigate through menus and buttons. With keyboard shortcuts, you can access commands and perform tasks much faster than you would be able to with a mouse or other input device.

In addition to speeding up your workflow, keyboard shortcuts can also help reduce the risk of repetitive strain injuries such as carpal tunnel syndrome. By reducing the amount of time you spend using a mouse or trackpad, you can help prevent the strain that can lead to these types of injuries.

Furthermore, keyboard shortcuts can also help you to stay focused on your work by reducing the amount of time you spend looking away from your screen. Since you can perform tasks quickly and easily with keyboard shortcuts, you can stay in the flow of your work and avoid distractions.

Go into the settings of many modern apps and you'll likely find a section on keyboard shortcuts. If there are shortcuts for the apps you use, it just makes sense to use them, but trying to learn them all at once isn't usually the way to go. Instead just take a few at a time, get good at using those, and then go back and learn a few more. Apps with great accessibility will allow you to navigate around the whole app and all of its functionality with your keyboard. Sholes would surely be proud.

Overall, getting comfortable with the keyboard and seeing it as a wieldy tool makes for a quick and generally enjoyable computer experience.

Some shortcuts that I find particularly useful in my work of writing software:

- [Vim](https://gist.github.com/tuxfight3r/0dca25825d9f2608714b)
- [VSCode](https://code.visualstudio.com/docs/getstarted/keybindings)
- [Vim for VSCode](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim)
- [MacOS](https://support.apple.com/en-us/HT201236)
- [Chrome](https://support.google.com/chrome/answer/157179?hl=en&co=GENIE.Platform%3DDesktop#zippy=)
- [Arc](https://www.hongkiat.com/blog/arc-browser-keyboard-shortcuts/)
- [Music](https://support.apple.com/guide/music/keyboard-shortcuts-mus1019/mac)


Mastering the keyboard is an invaluable skill that saves the most valuable currency...time.

Wield the Keyboard


Adventures can only start when the first step is taken.

The mark of a good developer is the ablity to share their learnings with others. It's 2023 and to no one's surprise, sharing online is the broadest platform one can share with.

So here we are, starting a blog, presenting my thoughts, works, learnings, and opinions with any who dare to endulge.

Enjoy


Queueing with BullMQ and Redis

Zooming & Panning

Virtual Lists are Beautiful

Wield the Keyboard

In The Beginning