Django models for storing web scrapper data

Element:
- Represents an HTML element.
- It has a name field, which is a character field with a maximum length of 200.
- Inherits from the Timestamped model (which is not shown here but probably contains timestamp fields like created_at and updated_at).
- Defines a custom string representation using the __str__ method.
Domain:
- Represents a domain.
- It has a name field, which is a character field with a maximum length of 200.
- Inherits from the Timestamped model.
- Defines a custom string representation using the __str__ method.
Page:
- Represents a web page.
- It has the following fields:
  - url: a URL field with a maximum length of 2048 characters.
  - domain: a foreign key to the Domain model, indicating the domain to which the page belongs.
  - pages: a many-to-many self-referential relationship to other Page instances, allowing pages to have children.
  - response: a text field for storing the page's content.
  - elements: a many-to-many relationship with the Element model through the PageElement intermediary model.
- Inherits from the Timestamped model.
- Defines a custom string representation using the __str__ method.
PageElement:
- Serves as an intermediary model between Page and Element to associate elements with pages and store additional information.
- It has the following fields:
  - page: a foreign key to the Page model.
  - element: a foreign key to the Element model.
  - value: a text field for storing additional information related to the element on the page.
Attribute:
- Represents attributes associated with an Element.
- It has the following fields:
  - element: a foreign key to the Element model, indicating the element to which the attribute belongs.
  - name: a character field for the attribute name with a maximum length of 255 characters.
  - value: a text field for storing the attribute's value.
- Inherits from the Timestamped model.
- Defines a custom string representation using the __str__ method.

python

from django.db import models

# Create your models here.

class Timestamped(models.Model):
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class Element(Timestamped):
    name = models.CharField(max_length=200)

    class Meta:
        default_related_name = 'elements'
        verbose_name = 'element'
        verbose_name_plural = 'elements'

    def __str__(self):
        return self.name
    

class Domain(Timestamped):
    name = models.CharField(max_length=200)

    class Meta:
        default_related_name = 'domains'
        verbose_name = 'domain'
        verbose_name_plural = 'domains'

    def __str__(self):
        return self.name



class Page(Timestamped):
    url = models.URLField(max_length=2048)
    domain = models.ForeignKey("Domain", on_delete=models.CASCADE)
    pages = models.ManyToManyField("self",symmetrical=False,related_name='children')
    response = models.TextField(null=True,blank=True)
    elements = models.ManyToManyField("Element",through="PageElement")


    class Meta:
        default_related_name = 'pages'
        verbose_name = 'page'
        verbose_name_plural = 'page'

    def __str__(self):
        return self.url
    

class PageElement(Timestamped):
    page = models.ForeignKey("Page", on_delete=models.CASCADE)
    element = models.ForeignKey("Element", on_delete=models.CASCADE)
    value = models.TextField(null=True,blank=True)


    

class Attribute(Timestamped):
    element = models.ForeignKey("Element", on_delete=models.CASCADE)
    name = models.CharField(max_length=255)
    value = models.TextField(null=True,blank=True)


    class Meta:
        default_related_name = 'attributes'
        verbose_name = 'attribute'
        verbose_name_plural = 'attributes'

    def __str__(self):
        return self.name

Related Posts

Django Message Mixin

Cms basic class based views

DRF Allow post put only to superuser

Get random object from django database model

Pass request get items to a form class in django

Django render ajax function