Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/hehe page #272

Merged
merged 22 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c1c77e6
add uploadedAt to Hehe type
Studsministern Apr 15, 2024
12a6b15
add photoUrl and PDF to PNG conversion to Hehe
Studsministern Apr 16, 2024
ce22a39
remove unnecessary async-await
Studsministern Apr 17, 2024
afaabc5
Merge branch 'main' into feature/hehe-page
Studsministern Apr 17, 2024
2e4b795
check that selected Hehe is a PDF
Studsministern Apr 17, 2024
08cc980
simplify endpoint URL
Studsministern Apr 17, 2024
e66a402
filter Hehe on FileType instead of string
Studsministern Apr 18, 2024
b6acee5
allow creation and removal of hehe covers
Studsministern Apr 24, 2024
de04cb8
change npm package that converts Hehe PDF to PNG
Studsministern Apr 29, 2024
e1d4f1e
make Hehe test cases functional
Studsministern Apr 29, 2024
d6cc4db
update CHANGELOG.md
Studsministern Apr 29, 2024
293cf47
remove unnecessary parameters for createHeheCover
Studsministern May 1, 2024
676fc81
add query to get hehes by pagination
Studsministern May 1, 2024
348a0f1
add pagination schema and improve Hehe pagination
Studsministern May 8, 2024
3398243
add max page size
Studsministern May 8, 2024
6bea96a
remake package-lock.json
Studsministern May 9, 2024
b094ef1
change docker image node version from 18-alpine to 18 in build step
Studsministern May 9, 2024
b1178b4
revert docker node version
Studsministern May 28, 2024
e3f0cb3
change pdf conversion from pdf-to-img package to microservice
Studsministern May 28, 2024
6947b0c
add example URL for pdf-to-png
Studsministern May 28, 2024
751e32f
fix example URL, change to base URL
Studsministern May 31, 2024
6417caa
Merge branch 'main' into feature/hehe-page
Studsministern May 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,7 @@ WIKI_PASSWORD=
WIKI_BASE_URL=https://wiki.esek.se

SKIP_ACCESS_CHECKS=false
POST_ACCESS_COOLDOWN_DAYS=90
POST_ACCESS_COOLDOWN_DAYS=90

# Pdf to png settings
PDF_TO_PNG_BASE_URL=https://pdf-to-png.esek.se
22 changes: 22 additions & 0 deletions CHANGELOG.MD
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,28 @@ Alla märkbara ändringar ska dokumenteras i denna fil.
Baserat på [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
och följer [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.10.0] - 2024-04-29

### Tillagt

- Attributen `coverEndpoint` och `coverId` för Hehe, där en bild för tidningens framsida genereras automatiskt utifrån PDFen med microservicen `pdf-to-png.esek.se`
- Exponerat attributet `uploadedAt` för Hehe, vilket är en `DateTime` för när Hehen laddades upp
- `DateTime` som en ny `Scalar`
- Schemas för pagination och paginerade Hehes
- Querien `paginatedHehes` för att hämta Hehes med paginering
- Utils-funktionen `createPageInfo` som skapar ett `PageInfo`-objekt för paginering
- Integrationstest för Hehe som kontrollerar att en bild kan skapas från en PDF
- Lagt till testfilen `test-hehe.pdf` som används i integrationstestet
- Enhetstest för Hehe som kontrollerar att en felaktig filtyp inte kan laddas upp
- Enhetstester för `paginatedHehes`
- Enhetstester för `createPageInfo`

### Ändrat

- `addHehe`-APIn så att denna dessutom skapar framsidan för tidningen och sparar motsvarande `coverId`
- Enhetstester och reducer-tester för Hehe så att dessa är kompatibla med tilläggen ovan
- Abstraherat ut uppladdning av filer i integrationstester till filen `fileUpload.ts`

## [1.9.0] - 2024-03-22
### Tillagt
- adds decibel_admin feature
Expand Down
14,747 changes: 1,671 additions & 13,076 deletions package-lock.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "ekorre-ts",
"version": "1.9.0",
"version": "1.10.0",
"description": "E-Sektionens backend",
"main": "src/index.ts",
"scripts": {
Expand Down
16 changes: 9 additions & 7 deletions prisma/schema.prisma
Original file line number Diff line number Diff line change
Expand Up @@ -213,13 +213,15 @@ model PrismaFile {
}

model PrismaHehe {
number Int
year Int
uploadedAt DateTime @default(now()) @map("uploaded_at")
uploader PrismaUser @relation(name: "PrismaHeheToPrismaUser", fields: [refUploader], references: [username])
refUploader String @map("ref_uploader")
file PrismaFile @relation(name: "PrismaFileToPrismaHehe", fields: [refFile], references: [id])
refFile String @unique @map("ref_file")
number Int
year Int
uploadedAt DateTime @default(now()) @map("uploaded_at")
uploader PrismaUser @relation(name: "PrismaHeheToPrismaUser", fields: [refUploader], references: [username])
refUploader String @map("ref_uploader")
file PrismaFile @relation(name: "PrismaFileToPrismaHehe", fields: [refFile], references: [id])
refFile String @unique @map("ref_file")
coverEndpoint String @map("cover_endpoint")
coverId String @map("cover_id")

// Year has more queries than number
@@id([year, number])
Expand Down
185 changes: 184 additions & 1 deletion src/api/hehe.api.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,27 @@
import config from '@/config';
import { NotFoundError, ServerError } from '@/errors/request.errors';
import { Logger } from '@/logger';
import { devGuard } from '@/util';
import { DEFAULT_PAGE_SIZE, MAX_PAGE_SIZE, createPageInfo, devGuard } from '@/util';
import { AccessType, FileType, Order, PageInfo, PaginationParams } from '@generated/graphql';
import { PrismaHehe } from '@prisma/client';
import axios from 'axios';
import { UploadedFile } from 'express-fileupload';
import FormData from 'form-data';
import { createReadStream } from 'fs';
import { writeFile } from 'fs/promises';
import path from 'path';

import FileAPI from './file.api';
import prisma from './prisma';

const {
FILES: { ENDPOINT, ROOT },
HEHES: { COVER_FOLDER },
PDF_TO_PNG,
} = config;

const fileApi = new FileAPI();

const logger = Logger.getLogger('HeheAPI');

export class HeheAPI {
Expand Down Expand Up @@ -59,23 +76,139 @@ export class HeheAPI {
return h;
}

/**
* Retrieves HeHEs by pagination, ordered by year and then number (in the inverted order)
* @param pagination The pagination parameters
* @returns A list containing the PageInfo and the PrismaHehe objects, which can then be reduced
*/
async getHehesByPagination(pagination?: PaginationParams): Promise<[PageInfo, PrismaHehe[]]> {
const page = pagination?.page ?? 1;
const pageSize = pagination?.pageSize ?? DEFAULT_PAGE_SIZE;
const order = pagination?.order ?? Order.Desc;

if (page < 1 || pageSize < 1) {
throw new ServerError('Sidnummer och HeHEs per sida måste vara större än 0');
}

if (pageSize > MAX_PAGE_SIZE) {
throw new ServerError(`Kan inte hämta fler än ${MAX_PAGE_SIZE} HeHEs per sida`);
}

const [count, hehes] = await prisma.$transaction([
prisma.prismaHehe.count(),
prisma.prismaHehe.findMany({
skip: (page - 1) * pageSize,
take: pageSize,
orderBy: [{ year: order }, { number: order === Order.Desc ? Order.Asc : Order.Desc }],
}),
]);

const pageInfo = createPageInfo(page, pageSize, count);

return [pageInfo, hehes];
}

/**
* Creates a cover image for a HeHE edition from a PDF
* @param uploaderUsername Username of the uploader
* @param fileId ID of the file containing the PDF
* @returns ID of the created cover image file
*/
async createHeheCover(uploaderUsername: string, fileId: string): Promise<string> {
const file = await fileApi.getFileData(fileId);

// If no file is provided
if (!file) {
logger.debug(`File ${fileId} can not be found`);
throw new NotFoundError('Filen kunde inte hittas, vilket kan bero på att den inte finns');
}

if (file.type !== FileType.Pdf) {
logger.debug('File is not a PDF');
throw new ServerError('Filen är inte en PDF');
}

// Get the PDF to convert
const pdfPath = `${ROOT}/${file.folderLocation}`;
const pdfStream = createReadStream(pdfPath);

// Add the PDF as a form-data object
const form = new FormData();
form.append('file', pdfStream);

// Convert the PDF to a PNG
const CONVERT_URL = PDF_TO_PNG.URL + '/convert';
const response = await axios
.create({
headers: form.getHeaders(),
responseEncoding: 'binary',
})
.post<ResponseType>(CONVERT_URL, form);

if (response.status !== 201) {
logger.debug('Could not convert PDF to image');
throw new ServerError('Kunde inte konvertera PDFen till en bild');
}

// Prepare values for the cover image
const pngBuffer = Buffer.from(response.data, 'binary');
const coverPath = `${path.parse(pdfPath).name}.png`;
const accessType = AccessType.Public;

// Creates the cover image as an UploadedFile and then saves it to the database
const uploadedFile = this.createUploadedFile(pngBuffer, coverPath, 'image/png');
const coverFile = await fileApi.saveFile(
uploadedFile,
accessType,
COVER_FOLDER,
uploaderUsername,
);

const coverId = coverFile.id;

if (coverId === '') {
logger.debug('Could not create cover image');
throw new ServerError('Kunde inte skapa omslagsbild');
}

return coverId;
}

/**
* Adds a new edition/paper of HeHE
* @param uploaderUsername Username of the uploader
* @param fileId ID of the file containing this paper
* @param coverId ID of the file containing the cover image of this paper
* @param number Number of the paper
* @param year What year the paper was published
*/
async addHehe(
uploaderUsername: string,
fileId: string,
coverId: string,
number: number,
year: number,
): Promise<boolean> {
const file = await fileApi.getFileData(fileId);

// If no file is provided
if (!file) {
logger.debug(`File ${fileId} can not be found`);
throw new NotFoundError('Filen kunde inte hittas, vilket kan bero på att den inte finns');
}

if (file.type !== FileType.Pdf) {
logger.debug('File is not a PDF');
throw new ServerError('Filen är inte en PDF');
}

try {
await prisma.prismaHehe.create({
data: {
refUploader: uploaderUsername,
refFile: fileId,
coverEndpoint: `${ENDPOINT}/${COVER_FOLDER}/`,
coverId,
number,
year,
},
Expand All @@ -99,6 +232,31 @@ export class HeheAPI {
* @param year What year the paper was published
*/
async removeHehe(number: number, year: number): Promise<boolean> {
const hehe = await prisma.prismaHehe.findFirst({
where: {
year,
number,
},
});

if (!hehe) {
logger.debug(`Could not find HeHE number ${number} for year ${year}`);
throw new ServerError(
'Kunde inte hitta upplagan av HeHE, vilket kan bero på att den inte finns',
);
}

// Try to remove the cover image
try {
await fileApi.deleteFile(hehe.coverId);
logger.info(`Deleted cover image for HeHE number ${number} for year ${year}`);
} catch (err) {
logger.error(err);
logger.error(
`Failed to remove existing cover image for HeHE number ${number} for year ${year}`,
);
}

try {
await prisma.prismaHehe.delete({
where: {
Expand All @@ -118,6 +276,31 @@ export class HeheAPI {
}
}

/**
* Creates an UploadedFile object from a buffer, for use with the file API
* @param data Buffer with the file's data
* @param name Name of the file
* @param type MIME type
* @returns
*/
private createUploadedFile(data: Buffer, name: string, type: string): UploadedFile {
const file: UploadedFile = {
name,
data,
size: data.byteLength,
encoding: '7bit',
tempFilePath: '',
truncated: false,
mimetype: type,
md5: '',
mv: async (newPath: string): Promise<void> => {
return writeFile(newPath, data);
},
};

return file;
}

/**
* Removes all HeHEs from the database.
*
Expand Down
18 changes: 18 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ const FILES = {
Number.parseInt(process.env.MAX_FILE_UPLOAD_SIZE_MB ?? '20') * BYTES_PER_MB, // Default 20 MB
};

/**
* Config for HeHEs
* @param {string} COVER_FOLDER - The folder to save HeHE covers in
*/
const HEHES = {
COVER_FOLDER: 'hehe-covers',
};

/**
* Config for Ebrev - our emailing service
* @param {string} URL - The base URL for Ebrevs API
Expand All @@ -44,6 +52,14 @@ const WIKI = {
PASSWORD: process.env.WIKI_PASSWORD ?? '',
};

/**
* Config for PDF to PNG conversion
* @param {string} URL - The base URL for the PDF to PNG microservice
*/
const PDF_TO_PNG = {
URL: process.env.PDF_TO_PNG_BASE_URL ?? '',
};

const JWT = {
SECRET: (process.env.JWT_SECRET as string) ?? '',
};
Expand All @@ -56,9 +72,11 @@ const config = {
SKIP_ACCESS_CHECKS: process.env.SKIP_ACCESS_CHECKS?.toLowerCase() === 'true',
POST_ACCESS_COOLDOWN_DAYS: Number.parseInt(process.env.POST_ACCESS_COOLDOWN_DAYS ?? '0'),
FILES,
HEHES,
EBREV,
LU,
WIKI,
PDF_TO_PNG,
JWT,
};

Expand Down
Loading
Loading