bj/anki-csv2ankicards
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

overhauled the project to get away from files (a little)

Browse Source
This commit is contained in:
B.J. Dweck 2023-09-11 19:09:18 +03:00
parent a13f92548c
commit c3bdf2dd5e
11 changed files with 247 additions and 243 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -2,3 +2,4 @@ venv/
*.pyc *.pyc
__pycache__/ __pycache__/
.env

27
.vscode/launch.json vendored Normal file
View File

@ -0,0 +1,27 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Flask",
"type": "python",
"request": "launch",
"module": "flask",
"envFile": "${workspaceFolder}/.env",
"env": {
"FLASK_APP": "server.py",
"FLASK_ENV": "development",
"FLASK_DEBUG": "0"
},
"args": [
"run",
"--no-debugger",
"--no-reload"
],
"jinja": true,
"justMyCode": true
}
]
}

128
README.md
View File

@ -1,26 +1,24 @@
# csv2ankicards # AnkiAI
A comprehensive toolkit that offers: AnkiAI is a robust system that converts images containing text into structured Anki cards using Optical Character Recognition (OCR) and OpenAI's GPT-4 language model. Users can quickly generate decks of flashcards from their images for effective study.
- Conversion of CSV files into Anki deck packages (.apkg files).
- Conversion of image files in a directory to a text file using Optical Character Recognition (OCR).
- Generation of CSV format question-answer pairs from textual content using OpenAI's GPT-3 model.
- **RESTful API endpoint to upload and convert multiple images directly into an Anki deck package.**
## Features ## Features
- Converts image content to textual content using OCR.
- Uses OpenAI's GPT-4 model to structure the content into Anki decks and cards.
- Outputs the structured content as an Anki package.
- Converts a CSV file with questions and answers into an Anki deck package. ## Dependencies
- Converts image files from a specified directory to a single text file using OCR. - genanki: Used for creating Anki decks and cards.
- Generates CSV formatted question-answer pairs based on a given text content, ideal for studying or summarization. - Pillow: Image processing library.
- For CSV: there are only two columns in the CSV file, separated by the first comma encountered. - openai: API library for OpenAI's GPT-4 model.
- CSV files should have a "Front" column for questions and a "Back" column for answers. - flask: Web server to host the service.
- **API endpoint that accepts multiple image uploads, processes them through the pipeline, and returns an Anki deck package.**
## Installation ## Setup and Installation
1. Clone this repository: 1. Clone this repository:
```bash ```bash
git clone https://git.rudefox.io/bj/anki-csv2ankicards.git git clone https://git.rudefox.io/bj/anki-json2ankicards.git
cd csv2ankicards cd json2ankicards
``` ```
2. Set up a virtual environment and activate it: 2. Set up a virtual environment and activate it:
@ -34,94 +32,36 @@ A comprehensive toolkit that offers:
pip install -r requirements.txt pip install -r requirements.txt
``` ```
## Configuration 4. Set up the OpenAI API key:
```bash
export OPENAI_API_KEY=your_openai_api_key
```
Before using the `text2csvdeck.py` script, ensure that you have set the `OPENAI_API_KEY` environment variable: 5. Run the server:
```bash
```bash python server.py
export OPENAI_API_KEY=your_openai_api_key_here ```
```
Remember to replace `your_openai_api_key_here` with your actual OpenAI API key.
## Usage ## Usage
### **REST API Usage** 1. Start the server as mentioned above.
To start the server: 2. Use a tool like [Postman](https://www.postman.com/) or `curl` to send images to `http://localhost:5000/deck-from-images` as a multi-part POST request.
```bash
python server.py
```
#### Endpoint: `/deck-from-images` 3. The server will respond with a downloadable Anki package. Import this into your Anki app and start studying!
**Method**: POST ## Modules
**Description**: Accepts multiple image uploads, processes them through the pipeline, and returns an Anki deck package. 1. **ankiai.py**: The main module that orchestrates the flow.
2. **images2text.py**: Converts image content into text using OCR.
3. **json2deck.py**: Converts structured JSON data into an Anki package.
4. **prompt4cards.py**: Uses OpenAI to structure the content into Anki decks and cards.
5. **server.py**: Flask server to host the service.
**Body**: ## Contributing
- form-data with key `image` and multiple image files as values. Contributions are welcome! Please submit a pull request or open an issue to discuss changes or fixes.
**Response**: ## License
- An `output.apkg` file ready for import into Anki. [MIT License](LICENSE)
### Pipeline Usage
To convert a directory of images directly to an Anki deck package:
```bash
python pipeline.py /path/to/your/image_directory/
```
This will process the images, extract text, convert text to a set of questions and answers in CSV format, and then produce an `output.apkg` file ready for import into Anki.
### Image to Text Conversion
To convert images from a directory to a single text file using OCR:
```bash
python images2text.py /path/to/your/image_directory/
```
This will produce a `final.txt` file which contains the text extracted from the images.
#### Supported Image Formats
Currently supported formats for the images are: `.png`, `.jpg`, and `.jpeg`.
### Text to CSV Deck Generation
To generate a CSV deck of question-answer pairs from a given text file:
```bash
python text2csvdeck.py /path/to/your/textfile.txt
```
This will analyze the content of the given text file and generate a corresponding `_deck.csv` file with questions and answers that capture the main points and themes of the text.
**Note:** This script uses the OpenAI GPT-3 model. Ensure you have the necessary API key and OpenAI Python client installed.
### CSV to Anki Conversion
To convert a CSV file into an Anki deck package:
```bash
python csv2ankicards.py /path/to/your/csvfile.csv output.apkg
```
This will produce an `output.apkg` file which can then be imported into Anki.
#### CSV Format
The CSV file should follow this format:
```
Front,Back
Your question here,Your answer here
Another question,list of: answer1, answer2, answer3
...
```
**Note:** If your answers contain commas, they will be considered as part of the answer. Only the first comma is used to separate the question from the answer.

21
ankiai.py Normal file
View File

@ -0,0 +1,21 @@
import sys
from images2text import main as ocr_images
from prompt4cards import prompt_for_card_content, response_to_json
from json2deck import to_package
def images_to_package(directory_path, outfile):
ocr_text = ocr_images(directory_path)
response_text = prompt_for_card_content(ocr_text)
deck_json = response_to_json(response_text)
to_package(deck_json).write_to_file(outfile)
print(f"Deck created at: {outfile}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python ankiai.py <directory_path_containing_images>")
sys.exit(1)
images_to_package(sys.argv[1])

49
csv2ankicards.py
View File

@ -1,49 +0,0 @@
import csv
import genanki
import sys
# Create a new model for our cards. This is necessary for genanki.
MY_MODEL = genanki.Model(
1607392319,
"Simple Model",
fields=[
{"name": "Question"},
{"name": "Answer"},
],
templates=[
{
"name": "Card 1",
"qfmt": "{{Question}}",
"afmt": "{{FrontSide}}<hr id='answer'>{{Answer}}",
},
])
def csv_to_anki(csv_path, output_path):
with open(csv_path, 'r', encoding='utf-8') as f:
reader = csv.reader(f)
# Skipping the header row
next(reader, None)
my_deck = genanki.Deck(2059400110, "CSV Deck")
for row in reader:
# Use row directly without splitting
question = row[0]
answer = ",".join(row[1:])
note = genanki.Note(
model=MY_MODEL,
fields=[question, answer]
)
my_deck.add_note(note)
genanki.Package(my_deck).write_to_file(output_path)
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: python convert.py <input_csv> <output_apkg>")
sys.exit(1)
input_csv = sys.argv[1]
output_apkg = sys.argv[2]
csv_to_anki(input_csv, output_apkg)
print(f"Deck created at: {output_apkg}")

2
images2text.py
View File

@ -80,7 +80,7 @@ def main(directory_path):
f.write("\n".join(final_text)) f.write("\n".join(final_text))
print(f"All images processed! Final output saved to {FINAL_OUTPUT}") print(f"All images processed! Final output saved to {FINAL_OUTPUT}")
return FINAL_OUTPUT # Add this line return final_text # Add this line
if __name__ == "__main__": if __name__ == "__main__":

55
json2deck.py Normal file
View File

@ -0,0 +1,55 @@
import json
import genanki
import sys
# Create a new model for our cards. This is necessary for genanki.
MY_MODEL = genanki.Model(
1607372319,
"Simple Model",
fields=[
{"name": "Title"},
{"name": "Question"},
{"name": "Answer"},
],
templates=[
{
"name": "{{Title}}",
"qfmt": "{{Question}}",
"afmt": "{{FrontSide}}<hr id='answer'>{{Answer}}",
},
])
def json_file_to_package(json_path):
with open(json_path, 'r', encoding='utf-8') as f:
json_data = json.load(f)
package = to_package(json_data)
return package
def to_package(deck_json):
deck_title = deck_json["DeckTitle"]
deck = genanki.Deck(1607372319, deck_title)
for card_json in deck_json["Cards"]:
title = card_json["Title"]
question = card_json["Question"]
answer = card_json["Answer"]
note = genanki.Note(
model=MY_MODEL,
fields=[title, question, answer]
)
deck.add_note(note)
return genanki.Package(deck)
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: python convert.py <input_json> <output_apkg>")
sys.exit(1)
input_json = sys.argv[1]
output_apkg = sys.argv[2]
json_file_to_package(input_json).write_to_file(output_apkg)
print(f"Deck created at: {output_apkg}")

27
pipeline.py
View File

@ -1,27 +0,0 @@
import sys
import os
from images2text import main as images_to_text
from text2csvdeck import text_file_to_csv_deck
CSV_DECK_NAME = "output_deck.csv"
APKG_NAME = "output.apkg"
def pipeline(directory_path):
# 1. Convert images in the directory to a text file
text_file_name = images_to_text(directory_path)
# 2. Convert the text file to a CSV deck using ChatGPT
text_file_to_csv_deck(text_file_name)
# 3. Convert the CSV deck to an Anki package
os.system(f"python csv2ankicards.py {CSV_DECK_NAME} {APKG_NAME}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python pipeline.py <directory_path_containing_images>")
sys.exit(1)
pipeline(sys.argv[1])

104
prompt4cards.py Normal file
View File

@ -0,0 +1,104 @@
import openai
import sys
import os
import json
CHAT_MODEL = "gpt-3.5-turbo"
OUTPUT_FILENAME = "output_deck.json"
API_KEY = os.environ.get("OPENAI_API_KEY")
if not API_KEY:
raise ValueError("Please set the OPENAI_API_KEY environment variable.")
openai.api_key = API_KEY
# Given prompt template
PROMPT_TEMPLATE = """
Please come up with a title for the deck and a set of 10 index cards for memorization,
including a title, front, and back for each card. The index cards should completely
capture the main points and themes of the text. In addition, they should contain any
numbers or data that humans might find difficult to remember. The goal of the index
card set is that one who memorizes it can provide a summary of the text to someone
else, conveying the main points and themes.
You will provide the deck title, and the titles, questions, and answers for each card
in a structured format as follows:
```
Deck Title: Title of the Deck
Cards:
- Title: Card Title 1
Front: What is the capital of New York?
Back: Albany
- Title: Card Title 2
Front: Where in the world is Carmen San Diego?
Back: Nobody knows
```
{content}
"""
def prompt_for_card_content(text_content):
# Prepare the prompt
prompt = PROMPT_TEMPLATE.format(content=text_content)
# Get completion from the OpenAI ChatGPT API
response = openai.ChatCompletion.create(
model=CHAT_MODEL,
messages=[
{"role": "user", "content": prompt}
],
temperature=0,
)
# Extract content from response and save to a new file
return response.choices[0]['message']['content']
def response_to_json(response_text):
lines = [line.strip() for line in response_text.split("\n") if line.strip()]
deck_title = None
cards = []
current_card = {}
for line in lines:
if "Deck Title:" in line and not deck_title:
deck_title = line.split("Deck Title:", 1)[1].strip()
elif "Title:" in line:
if current_card: # If there's a card being processed, add it to cards
cards.append(current_card)
current_card = {}
current_card["Title"] = line.split("Title:", 1)[1].strip()
elif "Front:" in line:
current_card["Question"] = line.split("Front:", 1)[1].strip()
elif "Back:" in line:
current_card["Answer"] = line.split("Back:", 1)[1].strip()
if current_card: # Add the last card if it exists
cards.append(current_card)
return {
"DeckTitle": deck_title,
"Cards": cards
}
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python text2jsondeck.py <text_file_path>")
sys.exit(1)
text_file_path = sys.argv[1]
# Read the text content
with open(text_file_path, 'r') as file:
text_content = file.read()
response_text = prompt_for_card_content(text_content)
deck_json = response_to_json(response_text)
with open(OUTPUT_FILENAME, 'w') as json_file:
json.dump(deck_json, json_file)
print(f"Saved generated deck to {OUTPUT_FILENAME}")

6
server.py
View File

@ -4,7 +4,7 @@ import os
import tempfile import tempfile
import shutil import shutil
from pipeline import pipeline from ankiai import images_to_package
app = Flask(__name__) app = Flask(__name__)
@ -16,9 +16,11 @@ def save_uploaded_images(images, directory):
for img in images: for img in images:
# Sanitize the filename # Sanitize the filename
safe_filename = secure_filename(img.filename) safe_filename = secure_filename(img.filename)
if not safe_filename: if not safe_filename:
# Handle the case where the filename becomes empty after sanitization # Handle the case where the filename becomes empty after sanitization
raise ValueError("Invalid filename") raise ValueError("Invalid filename")
filename = os.path.join(directory, safe_filename) filename = os.path.join(directory, safe_filename)
img.save(filename) img.save(filename)
@ -35,7 +37,7 @@ def deck_from_images():
save_uploaded_images(images, TEMP_DIR) save_uploaded_images(images, TEMP_DIR)
try: try:
pipeline(TEMP_DIR) images_to_package(TEMP_DIR, OUTPUT_FILE)
return send_from_directory('.', OUTPUT_FILE, as_attachment=True) return send_from_directory('.', OUTPUT_FILE, as_attachment=True)
except Exception as e: # Consider catching more specific exceptions except Exception as e: # Consider catching more specific exceptions
return jsonify({'error': str(e)}), 500 return jsonify({'error': str(e)}), 500

70
text2csvdeck.py
View File

@ -1,70 +0,0 @@
import openai
import sys
import os
CHAT_MODEL = "gpt-3.5-turbo"
OUTPUT_FILENAME = "output_deck.csv"
API_KEY = os.environ.get("OPENAI_API_KEY")
if not API_KEY:
raise ValueError("Please set the OPENAI_API_KEY environment variable.")
openai.api_key = API_KEY
# Given prompt template
PROMPT_TEMPLATE = """
Please come up with a set of 10 index cards for memorization, including front and back.
The index cards should completely capture the main points and themes of the text.
In addition, they should contain any numbers or data that humans might find difficult to remember.
The goal of the index card set is that one who memorizes it can provide a summary of the text to someone else, conveying the main points and themes.
You will provide the questions and answers to me in CSV format, as follows:
```
Front,Back
What is the capital of New York?,Albany
Where in the world is Carmen San Diego?,Nobody knows
```
The question/answer pairs shall not be numbered or contain any signs of being ordered.
{content}
"""
def text_file_to_csv_deck(text_file_path):
# Read the text content
with open(text_file_path, 'r') as file:
text_content = file.read()
content_to_csv(text_content)
def content_to_csv(text_content):
# Prepare the prompt
prompt = PROMPT_TEMPLATE.format(content=text_content)
# Get completion from the OpenAI ChatGPT API
response = openai.ChatCompletion.create(
model=CHAT_MODEL,
messages=[
{"role": "user", "content": prompt}
],
temperature=0,
)
# Extract CSV content from response and save to a new file
csv_content = response.choices[0]['message']['content']
with open(OUTPUT_FILENAME, 'w') as csv_file:
csv_file.write(csv_content)
print(f"Saved generated deck to {OUTPUT_FILENAME}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python text2csvdeck.py <text_file_path>")
sys.exit(1)
text_file_to_csv_deck(sys.argv[1])